TensorFlow Interview Questions (2026): 49 With Answers

A tf.Tensor is an immutable, multi-dimensional array, conceptually close to a NumPy array, but it carries extra machinery: it can live on GPU or TPU memory, it can participate in TensorFlow's automatic differentiation, and it can be part of a traced computation graph instead of running eagerly line by line. A NumPy array only ever lives on the CPU and has no built-in notion of gradients.

import tensorflow as tf
import numpy as np

t = tf.constant([[1, 2], [3, 4]])
n = np.array([[1, 2], [3, 4]])

print(t.numpy()) # converts back to a plain NumPy array
print(t.device)  # e.g. /job:localhost/replica:0/task:0/device:GPU:0

You can move between the two freely with .numpy() and tf.constant(), which is why most people don't think much about the distinction until they hit a performance question about where data actually lives.

Eager execution means operations run immediately when called, the same way normal Python code does, instead of building a symbolic graph first and running it later in a separate session. TensorFlow 1.x required you to build a graph and then call sess.run() to actually execute it, which was fast but famously hard to debug since print statements inside the graph didn't behave the way people expected.

TF 2.x flipped the default to eager because it matches how PyTorch worked and how most people already think about Python, line by line, with immediate errors and values you can inspect directly. Graphs didn't go away, they got moved behind @tf.function as an opt-in speed layer instead of the mandatory default.

tf.constant creates an immutable tensor, its value can never change after creation. tf.Variable creates mutable, stateful storage meant to persist and change across calls, which is exactly what a model's trainable weights need, since the optimizer has to update them in place after every training step.

w = tf.Variable(tf.zeros([10, 1])) # weights, updated during training
b = tf.constant(0.0)         # a fixed constant, never changes

Every layer's weights in Keras are tf.Variable objects underneath, which is also why model.trainable_variables is the exact list an optimizer needs to know what to update.

Sequential is a linear stack of layers, one input, one output, no branching, good for simple feedforward or CNN models. The Functional API lets you wire layers into an arbitrary directed graph, multiple inputs, multiple outputs, shared layers, skip connections, which covers most real architectures that aren't a straight stack. Subclassing tf.keras.Model gives you a plain Python class with a custom call() method, the most flexible option, needed when the forward pass has real conditional logic or a structure that doesn't map cleanly onto a static graph.

# Functional API, two inputs merged into one output
inputs_a = tf.keras.Input(shape=(10,))
inputs_b = tf.keras.Input(shape=(5,))
merged = tf.keras.layers.Concatenate()([inputs_a, inputs_b])
output = tf.keras.layers.Dense(1)(merged)
model = tf.keras.Model(inputs=[inputs_a, inputs_b], outputs=output)

compile() attaches the optimizer, the loss function, and any metrics to a model, and internally builds the training step's graph, but it doesn't do any training itself. Calling fit() afterward is what actually runs the forward pass, computes gradients, and updates weights for however many epochs you specify.

model.compile(
  optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
  metrics=["accuracy"],
)

A callback is an object with hooks that fire at specific points during training, start and end of an epoch, start and end of a batch, giving you a way to run code without modifying the training loop itself. EarlyStopping and ModelCheckpoint are the two you'll actually use in almost every real project.

callbacks = [
  tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=3, restore_best_weights=True),
  tf.keras.callbacks.ModelCheckpoint("best_model.keras", save_best_only=True),
]
model.fit(train_ds, validation_data=val_ds, epochs=50, callbacks=callbacks)

The Functional API treats every layer call as a node connecting tensors, not steps in a linear stack, so you can branch, merge, reuse the same layer instance on two different inputs, or expose more than one output from the same model. Sequential physically can't represent any of that, it only ever has one path from input to output.

shared_encoder = tf.keras.layers.Dense(32, activation="relu")
out_a = shared_encoder(input_a) # same weights
out_b = shared_encoder(input_b) # reused on a different input

Multi-output models, siamese networks, and anything with a skip connection all need the Functional API or subclassing for exactly this reason.

TensorFlow raises a RuntimeError telling you the tape's resources have already been released. A non-persistent tape is designed to be used exactly once, and it frees the recorded operations as soon as the first tape.gradient() call finishes, since keeping them around for every single training step would waste memory nobody needed.

tf.data builds an input pipeline that can shuffle, batch, prefetch, and preprocess data on a separate thread while the GPU is still busy training on the previous batch, so the accelerator doesn't sit idle waiting for the next batch to load. Feeding raw NumPy arrays directly means the entire dataset has to fit in memory and there's no overlap between data loading and computation.

dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.shuffle(10000).batch(32).prefetch(tf.data.AUTOTUNE)

TFRecord is a binary, sequential storage format built for streaming large datasets efficiently off disk, rather than opening thousands of individual small files, which is slow on most filesystems and especially slow on network-attached storage. Data gets serialized into protobuf-based records once, ahead of training, so the training loop just reads a sequential binary stream instead of parsing images or JSON on every single epoch.

def serialize_example(feature, label):
  feature_dict = {
    "feature": tf.train.Feature(float_list=tf.train.FloatList(value=feature)),
    "label": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
  }
  return tf.train.Example(features=tf.train.Features(feature=feature_dict)).SerializeToString()

SavedModel is TensorFlow's own format: a directory containing the graph, the weights, and the computation itself as a serialized graph, which means it can be loaded and run without the original Python class definition at all, and it's what TensorFlow Serving and TensorFlow Lite conversion expect natively. HDF5 saves the architecture as JSON and the weights as a binary blob, but reloading it still needs the original model class or config available in your Python environment.

model.save("my_model")      # SavedModel directory, framework-native
model.save("my_model.h5")    # HDF5 file, needs the class definition to reload cleanly

Keras's newer native.keras format sits in between, a single zipped file that still preserves full architecture and optimizer state.

enable_check_numerics inserts checks after every operation that can produce NaN or infinity, and raises an error immediately at the operation that first produced the bad value, instead of letting it silently propagate for dozens of steps before the loss finally shows up as NaN with no clue which layer caused it.

tf.debugging.enable_check_numerics()

It's genuinely useful in development when you're chasing down exactly where a NaN first appears. You wouldn't leave it enabled in production, it adds real overhead to every single operation, since it's meant as a diagnostic tool for tracking down a specific bug, not something you run permanently in a live training job.

TFX (TensorFlow Extended) is a production ML pipeline framework that chains together the full lifecycle around a model, data validation, feature transformation, training, model evaluation against a baseline, and pushing to a serving destination, as a reproducible, orchestrated pipeline rather than a collection of loose scripts someone runs by hand in a particular order.

A plain training script trains a model. It doesn't validate that new incoming data hasn't silently drifted from what the model was trained on, doesn't automatically compare a freshly trained model against the currently deployed one before promoting it, and doesn't give you a repeatable, versioned record of exactly what produced a given model artifact. TFX exists specifically to close that gap for teams retraining models on a recurring schedule against live data.

@tf.function traces your Python function once, recording every TensorFlow operation it calls into a static computation graph, and then replaces subsequent calls with that compiled graph instead of re-running the Python code. The graph version skips Python's interpreter overhead on every call and can apply graph-level optimizations, constant folding, operation fusion, that eager mode can't.

@tf.function
def train_step(x, y):
  with tf.GradientTape() as tape:
    preds = model(x)
    loss = loss_fn(y, preds)
  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))
  return loss

The catch is that tracing only happens for the shapes and dtypes it first sees, which is exactly what makes retracing such a common interview follow-up.

A regular Python list works fine in eager mode, but inside a traced graph it doesn't exist as a graph-native concept, so appending to a plain list inside a @tf.function loop either fails or only captures the first trace's behavior. tf.TensorArray is the graph-compatible equivalent: a dynamically-sized, write-once-per-index array that AutoGraph knows how to represent as a real graph operation.

@tf.function
def cumulative_sum(x):
  arr = tf.TensorArray(dtype=tf.float32, size=tf.shape(x)[0])
  total = 0.0
  for i in tf.range(tf.shape(x)[0]):
    total += x[i]
    arr = arr.write(i, total)
  return arr.stack()

You mostly need this for genuinely dynamic-length loops inside graph mode, custom RNN cells written by hand, beam search decoding, that sort of thing. Most day-to-day model code never touches TensorArray directly because Keras layers already handle it internally.

AutoGraph rewrites Python-level if, for, and while statements into their graph-native equivalents, tf.cond and tf.while_loop, so that branches depending on tensor values still work correctly once traced into a static graph, rather than only ever taking whichever branch was true during the original trace.

@tf.function
def classify(x):
  if tf.reduce_sum(x) > 0:
    return "positive"
  else:
    return "non-positive"

If the condition depends only on a Python value known at trace time, a hyperparameter flag, say, AutoGraph doesn't need to convert it at all, it just bakes in whichever branch was taken. The conversion only kicks in when the condition depends on an actual tensor whose value isn't known until runtime.

AutoGraph is the source-to-source transformation TensorFlow runs automatically whenever you apply @tf.function. It parses your Python function's abstract syntax tree and rewrites control-flow statements and certain Python constructs into TensorFlow graph operations, so you can write normal-looking Python and still get a graph that handles data-dependent branching correctly.

It doesn't convert everything. Arbitrary Python side effects, mutating a global list, calling a non-TensorFlow library inside the loop, don't get captured into the graph and will only run once, during tracing, not on every subsequent call. That's the single most common source of "my code works in eager mode but produces wrong results once I add @tf.function" bugs.

Subclass tf.keras.layers.Layer, define your trainable weights, and implement call() to describe the forward pass. Weights created inside build() get automatically tracked by Keras so they show up in model.trainable_variables and get saved with the model.

class LinearLayer(tf.keras.layers.Layer):
  def __init__(self, units):
    super().__init__()
    self.units = units

  def build(self, input_shape):
    self.w = self.add_weight(
      shape=(input_shape[-1], self.units), initializer="glorot_uniform"
    )
    self.b = self.add_weight(shape=(self.units,), initializer="zeros")

  def call(self, inputs):
    return tf.matmul(inputs, self.w) + self.b

__init__() runs once, when you instantiate the layer, before it's ever connected to any input, so it doesn't know the input's shape yet. build() runs lazily, the first time the layer is actually called with real input, and by then Keras can pass in the input shape, which is what lets you size your weight matrices correctly without the caller having to specify the input dimension up front.

The practical benefit is that a layer like Dense(64) can be created without knowing what feeds into it, and Keras figures out the input dimension automatically the first time data actually flows through. Skipping build() and creating weights directly in __init__() only works if you already know the input shape at construction time, which isn't always true.

You open a GradientTape, run the forward pass inside it, compute the loss, pull the gradients with respect to the model's trainable variables, and apply them through the optimizer yourself. Wrapping the whole step in @tf.function gets most of the speed back that model.fit() gives you for free.

@tf.function
def train_step(x, y):
  with tf.GradientTape() as tape:
    preds = model(x, training=True)
    loss = loss_fn(y, preds)
  grads = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))
  return loss

for epoch in range(epochs):
  for x_batch, y_batch in train_dataset:
    loss = train_step(x_batch, y_batch)

People reach for this when they need something fit() doesn't expose cleanly: multiple losses computed on different schedules, custom gradient clipping logic, or a GAN-style setup where two models need to be updated in a specific alternating order.

Inside a GradientTape context, TensorFlow records every differentiable operation applied to a watched tensor, building a small computation graph of just that forward pass. Calling tape.gradient() then walks that recorded graph backward, applying the chain rule operation by operation, to compute the derivative of your output with respect to whatever inputs you ask for.

x = tf.Variable(3.0)
with tf.GradientTape() as tape:
  y = x ** 2
dy_dx = tape.gradient(y, x) # 6.0

Once tape.gradient() runs, the recorded operations get released by default, which is exactly why calling it a second time on the same tape raises an error unless you asked for a persistent tape up front.

tf.Variable objects get watched automatically because they're assumed to be trainable by default. Plain tensors, tf.constant values or inputs that aren't variables, don't get watched automatically, since recording every single tensor operation would be wasteful when most of them are never actually needed for a gradient. You call tape.watch() explicitly when you need a gradient with respect to something that isn't a Variable, most commonly the input itself, for adversarial example generation or saliency maps.

x = tf.constant(3.0)
with tf.GradientTape() as tape:
  tape.watch(x)
  y = x ** 2
dy_dx = tape.gradient(y, x) # 6.0, would be None without tape.watch(x)

By default, a GradientTape releases the resources it used to record the forward pass the first time you call tape.gradient(), which keeps memory usage low but means you only get one gradient computation per tape. Setting persistent=True keeps the recorded graph around so you can call tape.gradient() multiple times against different targets, useful when you need gradients of two separate losses with respect to the same set of variables.

with tf.GradientTape(persistent=True) as tape:
  y1 = x ** 2
  y2 = x ** 3

dy1 = tape.gradient(y1, x)
dy2 = tape.gradient(y2, x)
del tape # release manually once you're done, since Python's GC won't hurry

You do have to manually delete the tape once you're finished with it, since it holds onto memory until you do, and letting it sit around unused in a long training loop is a real, if subtle, memory leak.

prefetch overlaps the preparation of the next batch with the training step currently running on the current batch, using a separate background thread, so the GPU doesn't stall waiting on the CPU to finish the next batch's preprocessing. AUTOTUNE tells TensorFlow to dynamically tune the prefetch buffer size at runtime based on available resources, instead of you guessing a fixed number.

dataset = dataset.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)
dataset = dataset.batch(32).prefetch(tf.data.AUTOTUNE)

Without it, the profiler timeline shows visible gaps where the GPU is idle waiting on input, exactly the kind of bottleneck interviewers like asking candidates to diagnose from a hypothetical trace.

.map() applies a transformation to each element of a single dataset in order..interleave() reads from multiple source datasets, files, typically, concurrently and interleaves their elements together, which is what you actually want when you have many separate TFRecord shards and don't want to read them strictly one after another.

files = tf.data.Dataset.list_files("data/shard-*.tfrecord")
dataset = files.interleave(
  tf.data.TFRecordDataset, cycle_length=4, num_parallel_calls=tf.data.AUTOTUNE
)

Using interleave with a decent cycle_length also helps randomize which shard's data lands in a given batch, which matters if the shards themselves aren't already shuffled at the file level.

.map() before.batch() applies your transformation to one element at a time, which is simpler to write but leaves parallelism entirely up to num_parallel_calls on individual elements..batch() before.map() applies your function to an already-batched tensor, which lets vectorized operations run once across the whole batch instead of once per element, often noticeably faster for anything that vectorizes cleanly, image resizing, normalization, numeric feature scaling.

# Vectorized mapping, operates on a batch tensor at once
dataset = dataset.batch(32).map(lambda x, y: (x / 255.0, y))

The trade-off is that per-element logic that genuinely needs to differ row by row, variable-length sequence padding logic, for instance, is awkward or impossible to express once everything's already batched.

A signature defines the named input and output tensors a SavedModel exposes for inference, essentially a typed function contract that TensorFlow Serving or any other consumer can call without knowing anything about the model's internal architecture. Without an explicit signature, a serving system has no reliable way to know what shape of input the model expects or what to call the output it gets back.

@tf.function(input_signature=[tf.TensorSpec(shape=[None, 224, 224, 3], dtype=tf.float32)])
def serving_fn(images):
  return {"predictions": model(images)}

tf.saved_model.save(model, "export_path", signatures={"serving_default": serving_fn})

You run the model through the TFLiteConverter, which takes a SavedModel or a Keras model and produces a.tflite flatbuffer optimized for mobile and embedded inference. The conversion can fold in quantization, and it strips out training-only operations, since a.tflite model is inference-only, it has no concept of gradients or an optimizer at all.

converter = tf.lite.TFLiteConverter.from_saved_model("export_path")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
with open("model.tflite", "wb") as f:
  f.write(tflite_model)

Not every op TensorFlow supports has a TFLite equivalent, so a model built with an exotic custom layer sometimes needs a custom TFLite op registered by hand or a fallback to the TensorFlow Lite Select TF ops runtime, which pulls in a much larger binary.

Quantization reduces the numerical precision of a model's weights and activations, typically from 32-bit floats down to 8-bit integers, which shrinks the model's size roughly fourfold and speeds up inference substantially on hardware with efficient integer math, mobile CPUs especially. The trade-off is a small, usually acceptable, drop in accuracy, since you're throwing away precision the network was originally trained with.

converter.representative_dataset = representative_data_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]

Post-training quantization is the easy path, run it on an already-trained model with a small representative dataset. Quantization-aware training simulates the precision loss during training itself, which usually recovers more of the accuracy gap at the cost of a more involved training setup.

TF Serving expects a directory structure where each subdirectory is a numbered version of the SavedModel, and it automatically loads whichever version number is highest by default, while still being able to serve older versions on request if configured to keep them loaded. Rolling out a new model is often as simple as dropping a new numbered folder next to the old ones, no restart needed, since Serving polls for new versions.

# export_path/
#  1/ (older SavedModel)
#  2/ (new SavedModel, gets picked up automatically)

This is exactly what makes canary rollouts and instant rollback possible: point traffic at version 1 again if version 2 turns out to regress a metric in production, no redeploy required.

MirroredStrategy replicates the entire model onto every GPU on a single machine, feeds each replica a different slice of the current batch, and after the forward and backward pass, synchronously averages gradients across all replicas using an all-reduce operation before applying the update, so every replica's weights stay identical after every step.

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
  model = build_model()
  model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")

model.fit(train_dataset, epochs=10)

Everything that creates variables, model construction, optimizer creation, has to happen inside strategy.scope(), which is the single most common source of bugs candidates describe running into the first time they try this.

MirroredStrategy synchronizes replicas that live on GPUs inside one physical machine. MultiWorkerMirroredStrategy extends the same synchronous, all-reduce idea across multiple separate machines on a network, which means it also has to handle cluster configuration, each worker needs to know its own role and the addresses of every other worker, via the TF_CONFIG environment variable, and network communication overhead becomes a real factor instead of something you can ignore.

import os, json
os.environ["TF_CONFIG"] = json.dumps({
  "cluster": {"worker": ["host1:port", "host2:port"]},
  "task": {"type": "worker", "index": 0},
})
strategy = tf.distribute.MultiWorkerMirroredStrategy()

Data parallelism keeps a full copy of the model on every device and splits the batch across them, which is what MirroredStrategy and MultiWorkerMirroredStrategy both do. Model parallelism instead splits the model itself, different layers or different parts of one layer, across devices, which you need when a single model no longer fits in one device's memory at all, regardless of batch size.

TensorFlow's mainstream distribute strategies are almost entirely data-parallel. Model parallelism in TensorFlow is far less turnkey and typically means manually placing operations on specific devices with tf.device(), or reaching for a purpose-built library rather than the built-in distribute API, which is exactly why it comes up less often outside of teams training genuinely huge models.

MirroredStrategy splits whatever batch size you configure across all replicas, so a batch size of 64 across 4 GPUs means each GPU actually only processes 16 examples per step. Forgetting this and keeping the same per-GPU batch size you used on a single GPU means the effective global batch size silently balloons, which changes training dynamics and often the learning rate you actually need.

global_batch_size = per_replica_batch_size * strategy.num_replicas_in_sync
dataset = dataset.batch(global_batch_size)

The fix is always to think in terms of global batch size explicitly and scale it, and usually the learning rate alongside it, on purpose as you add replicas, rather than assuming the framework will keep the effective batch size constant for you.

Mixed precision runs most operations in float16 (or bfloat16 on TPUs), which is faster and uses less memory on modern GPU tensor cores, while keeping certain numerically sensitive operations, weight updates in particular, in float32 to preserve stability. TensorFlow's official mixed precision guide covers the policy API that makes this close to a one-line change for most models.

from tensorflow.keras import mixed_precision
mixed_precision.set_global_policy("mixed_float16")

model = build_model() # layers now compute in float16, keep float32 variables

XLA (Accelerated Linear Algebra) is a compiler that fuses multiple TensorFlow operations into fewer, larger optimized kernels, reducing the overhead of launching many small GPU kernels back to back and often cutting memory traffic between them. It's the same compiler TPUs rely on by default, and on GPU it's opt-in.

@tf.function(jit_compile=True)
def train_step(x, y):
 ...

It doesn't help every model equally. Models dominated by a few large matrix multiplications see less benefit than models with lots of small, varied operations that XLA can meaningfully fuse together, and the very first call under jit_compile also pays a noticeable one-time compilation cost.

The TensorFlow Profiler, accessible through TensorBoard, captures a timeline of exactly what the CPU and GPU are each doing during a slice of training, and its "input pipeline analyzer" view specifically flags whether the GPU is spending significant time idle waiting on input data versus genuinely compute-bound.

tf.profiler.experimental.start("logdir")
# run a few training steps
tf.profiler.experimental.stop()

A model that's input-bound needs work on the tf.data pipeline, more parallelism in.map(), better prefetching. A model that's genuinely compute-bound on the GPU needs a different fix entirely, mixed precision, XLA, or a smaller or more efficient architecture. Guessing which one you're dealing with without actually profiling first wastes a lot of time chasing the wrong fix.

Both frameworks run eagerly by default now, so day-to-day model-writing code looks fairly similar in either. The real difference shows up at the compile step: TensorFlow's @tf.function is an explicit, separate decision you opt into for a specific function, while PyTorch's torch.compile applies more broadly with less code restructuring, and PyTorch's dynamic graph tends to tolerate genuinely dynamic, data-dependent control flow with fewer surprises than tracing under AutoGraph does.

TensorFlow still tends to have the edge on production serving tooling, TensorFlow Serving and the TFX ecosystem are more mature than most PyTorch-native equivalents, which is a real, practical reason some teams keep TensorFlow around even as their research workflows have moved elsewhere.

Keras 3 rewrote Keras as a framework-agnostic API that can run on top of TensorFlow, PyTorch, or JAX as interchangeable backends, instead of being TensorFlow-only. You write the same model code and swap the backend with an environment variable, which means a team can prototype in JAX for speed, train in TensorFlow for its data pipeline tooling, or deploy through whichever backend has the best serving story, without rewriting the model itself.

import os
os.environ["KERAS_BACKEND"] = "torch" # or "tensorflow", or "jax"
import keras

It matters for this conversation because it quietly dissolves part of the TensorFlow-versus-PyTorch framing at the model-authoring layer. The meaningful differences increasingly live in the ecosystem around each backend, deployment tooling, hardware support, data pipelines, not in how you write the model itself anymore.

TensorFlow still makes sense when the deployment target is mobile or embedded and you need TensorFlow Lite's maturity, when the team already runs TensorFlow Serving infrastructure and doesn't want to stand up a second serving stack, or when the project needs TPU access at scale, since TPU support has historically been deeper and more battle-tested in the TensorFlow ecosystem than elsewhere.

Outside of those specific cases, plenty of teams pick TensorFlow simply because it's what the existing codebase and the team's institutional knowledge already runs on, and rewriting a working production pipeline in a different framework rarely earns its cost on its own. That's a defensible answer too, and honestly a more common real-world reason than any of the technical ones.

Retracing happens when a @tf.function-decorated function gets called with an input signature it hasn't seen before, a new shape, a new dtype, or a Python scalar with a different value passed by reference instead of as a tensor. TensorFlow has to build a brand new graph for that signature, which costs real time, and if your training loop keeps feeding it slightly different shapes, it retraces on almost every call instead of reusing one compiled graph.

@tf.function
def add(a, b):
  return a + b

add(tf.constant(1), tf.constant(2))   # traces once
add(tf.constant(1.0), tf.constant(2.0)) # retraces, different dtype
add(1, 2)                # retraces again, Python ints, not tensors

The silent part is that it still produces the correct answer every time, it just quietly gets slower and slower, and nothing raises an error to tell you why. Passing a variable batch size as a Python int instead of a tensor is the single most common real-world cause I've seen. Setting input_signature explicitly on the tf.function pins the expected shapes and dtypes so retracing can't happen silently.

You nest one GradientTape inside another. The inner tape records the first derivative, and since that gradient computation itself happens inside the outer tape's recording scope, the outer tape can differentiate it again to get the second derivative.

x = tf.Variable(3.0)
with tf.GradientTape() as outer_tape:
  with tf.GradientTape() as inner_tape:
    y = x ** 3
  dy_dx = inner_tape.gradient(y, x) # first derivative: 3x^2 = 27
d2y_dx2 = outer_tape.gradient(dy_dx, x) # second derivative: 6x = 18

This shows up in interviews for physics-informed neural networks and certain optimization algorithms that need curvature information, not just the gradient itself. Candidates who've only ever trained a standard classifier rarely have a reason to have hit this pattern before, which is exactly why it's a good filter question at the senior level.

tf.custom_gradient lets you override how automatic differentiation computes the backward pass for a specific operation, instead of relying on TensorFlow deriving it automatically from the forward computation. You need it when the mathematically correct gradient isn't the one you actually want, straight-through estimators for quantization, or when the automatic derivative is numerically unstable and you know a more stable closed-form alternative.

@tf.custom_gradient
def clip_gradient(x, clip_value):
  def grad(dy):
    return tf.clip_by_value(dy, -clip_value, clip_value), None
  return tf.identity(x), grad

The straight-through estimator for quantization-aware training is the textbook example: the forward pass rounds a value to a discrete step, which has zero gradient almost everywhere, so the custom backward pass just passes the incoming gradient through unchanged instead of the true, useless derivative.

The safest approach is to shard at the file level before any shuffling or repeating happens, giving each worker a disjoint set of files, using Dataset.shard(num_workers, worker_index) or by splitting the file list itself. Sharding after a.repeat() call is a common bug, since it can cause different workers to see overlapping or duplicated data across epochs instead of a clean, non-overlapping split.

dataset = tf.data.Dataset.list_files("data/shard-*.tfrecord")
dataset = dataset.shard(num_shards=num_workers, index=worker_index)
dataset = dataset.interleave(tf.data.TFRecordDataset)
dataset = dataset.shuffle(1000).repeat().batch(32)

tf.distribute strategies actually handle a lot of this automatically through experimental_distribute_dataset, but understanding what's happening manually is exactly what separates someone who's debugged a real data-skew bug from someone repeating documentation.

model.save() is the Keras-level convenience API. It saves the model along with Keras-specific metadata, the optimizer state, compiled loss and metrics, so you can resume training exactly where you left off. tf.saved_model.save() is the lower-level TensorFlow API that any tf.Module can use, not just Keras models, and it saves whatever concrete functions and signatures you explicitly attach, without any assumption that what you're saving is a trainable Keras model at all.

The practical difference shows up when you're exporting a model purely for inference: tf.saved_model.save with a custom serving signature gives you tighter control over exactly what gets exposed to a serving system, versus model.save(), which bakes in the full Keras training-oriented object graph whether you need it for inference or not.

TPUStrategy targets Google's TPU accelerators, and because TPUs compile the entire training step ahead of time into a fixed, static computation graph, it's far less forgiving of dynamic shapes or Python-level control flow than a GPU strategy is. Every input batch has to have a fixed, known shape, ragged tensors and variable-length sequences without explicit padding will fail or force expensive recompilation, and data has to come from a tf.data pipeline the TPU can pull from directly rather than being fed from local Python memory.

resolver = tf.distribute.cluster_resolver.TPUClusterResolver()
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.TPUStrategy(resolver)

Candidates who've only trained on GPUs are frequently surprised the first time a dynamic-shape model that ran fine on a GPU throws a cryptic shape-related error the moment it's pointed at a TPU. That surprise itself is a fair thing for an interviewer to probe, since it reveals whether someone actually understands what makes TPUs fast in the first place.

float16 has a much smaller dynamic range than float32, so small gradient values that would be perfectly representable in float32 can underflow to zero in float16, and the reverse can happen too, values overflowing to infinity. Once a NaN or infinity shows up anywhere in the gradient computation, it propagates through every subsequent operation and the entire loss goes to NaN.

optimizer = tf.keras.optimizers.Adam(1e-3)
optimizer = mixed_precision.LossScaleOptimizer(optimizer) # wraps the optimizer

The fix is loss scaling: multiply the loss by a large constant before backpropagating, which pushes small gradient values up into float16's representable range, then divide the resulting gradients back down by that same factor before applying the update. Keras's LossScaleOptimizer does this automatically, including dynamically adjusting the scale factor if it detects overflow, which is why you almost never write loss scaling logic by hand anymore.

Shape mismatches that only surface deep inside a traced @tf.function are a classic one, the error message often points at an internal TensorFlow op several layers removed from the line of your code that actually caused it, since the traceback reflects the compiled graph's structure, not your original Python call stack in a way that's always obvious.

tf.config.run_functions_eagerly(True) # temporarily disable graph tracing for debugging

The practical fix is to temporarily force eager execution with run_functions_eagerly(True), which gets you normal Python tracebacks and lets you use pdb or print statements exactly where the problem actually is, then flip it back off once you've found and fixed the bug. Debugging inside an actual graph is painful enough that almost nobody does it directly if there's an eager-mode alternative available.

For years, tf.keras was TensorFlow's bundled, TensorFlow-specific fork of Keras, while a separately installed keras package on PyPI was an older, standalone, multi-backend version that predated Keras 3. Code written against one didn't always behave identically to code written against the other, and pinning the wrong keras version alongside a given TensorFlow version was a genuinely common source of import errors and subtly mismatched behavior in real projects.

import tensorflow as tf
print(tf.keras.__version__) # tf.keras, bundled with this TensorFlow install

import keras
print(keras.__version__)   # standalone keras package, may be Keras 3 or older

Since Keras 3 shipped as the standalone package with TensorFlow as one of its supported backends, the two paths have converged again for the versions people actually install today, but candidates who worked through the transition years remember exactly why "which keras is this" used to be a genuinely annoying question to answer on a new machine.