Databricks Cannot Find Your Iceberg Table in Glue — The Catalog Configuration That Fails Silently

A Databricks engineer registers an Iceberg table in AWS Glue, confirms the table exists in the Glue console, runs SHOW TABLES IN glue_catalog.my_database from a Databricks notebook, and gets back an empty result. Or worse: AnalysisException: [TABLE_OR_VIEW_NOT_FOUND]. The table is there. Glue knows about it. S3 has the data files and the metadata directory. But Spark cannot see it.

The root cause is almost always a Spark catalog misconfiguration — but the error message gives no signal about which property is wrong or missing. Spark's catalog resolution treats a misconfigured external catalog the same way it treats a catalog that doesn't exist: it falls back to the default spark_catalog, searches Hive metastore or Unity Catalog, finds nothing, and reports the table as missing.

This happens because Databricks clusters don't ship with the Iceberg catalog classes on the classpath by default. You're bolting on an external catalog at runtime through Spark properties and a Maven-coordinate JAR. If any piece of that chain is broken — wrong class name, missing JAR, absent IAM policy — the failure mode is the same silent nothing.

The Databricks Community thread on this topic drew seven replies, most of them posting slightly different property combinations. That pattern alone tells you something: the configuration surface is narrow but unforgiving. There are six properties that must all be correct simultaneously, and Spark will not tell you which one failed.

Here is the complete set of Spark configuration properties required to read an Iceberg table through AWS Glue Data Catalog on a Databricks cluster:

`` spark.sql.extensions = org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.catalog.glue_catalog = org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.glue_catalog.catalog-impl = org.apache.iceberg.aws.glue.GlueCatalog spark.sql.catalog.glue_catalog.warehouse = s3://your-bucket/warehouse/ spark.sql.catalog.glue_catalog.io-impl = org.apache.iceberg.aws.s3.S3FileIO ``

Plus the Iceberg runtime JAR: org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.0 (adjust the version suffix for your Spark and Scala versions).

Each property fails differently when wrong. Omit catalog-impl and Spark falls back to a Hadoop catalog that looks for metadata in warehouse path directly, bypassing Glue entirely. Omit io-impl and Iceberg defaults to HadoopFileIO, which works for HDFS but throws java.lang.InstantiationException or silent read failures on S3 paths. Set warehouse to an S3 path without a trailing slash and some Iceberg versions will fail to resolve the metadata pointer.

The runtime JAR version matters more than you'd expect. Spark 3.5 clusters need iceberg-spark-runtime-3.5_2.12. Use 3.4 and the classes load but method signatures won't match, producing NoSuchMethodError at query time — not at cluster startup. Set the JAR through the cluster's Libraries tab or an init script, not through %pip install, which won't place it on the Spark classpath correctly.

The glue_catalog name in the property keys is arbitrary — call it ice, glue, or prod_iceberg. But whatever name you choose, every subsequent property must use the same prefix. A typo in the prefix (e.g., spark.sql.catalog.glue.catalog-impl vs. spark.sql.catalog.glue_catalog.catalog-impl) creates a second, incomplete catalog definition that Spark ignores without warning.

Even with every Spark property correct, the query will fail if the Databricks cluster's IAM role lacks the right Glue and S3 permissions. The deceptive part: Spark registers the catalog successfully during session initialization. The SHOW DATABASES IN glue_catalog command might even work, because listing databases only requires glue:GetDatabases. The failure surfaces later, when you try to read a specific table.

Reading an Iceberg table through Glue requires at minimum: glue:GetTable, glue:GetDatabase, s3:GetObject, s3:ListBucket, and s3:GetBucketLocation on both the warehouse bucket and the actual data location (which may differ if the table was created with a custom location property). If the table uses partition specs, add glue:GetPartitions.

The IAM error messages are marginally better than Spark's. You'll see AccessDeniedException from the Glue SDK or 403 Forbidden from S3. But these exceptions get wrapped in Spark's AnalysisException, and depending on the Databricks Runtime version, the original AWS error may be truncated in the notebook output. Check the Spark driver logs (accessible through the cluster's Log Delivery or the Spark UI) for the full stack trace.

A common misstep: the cluster's instance profile has Glue permissions scoped to a specific AWS account, but the Iceberg table references a cross-account Glue database. Cross-account Glue access requires a resource policy on the target Glue catalog and glue:GetTable permission in the source account's IAM policy with the target catalog ARN. Miss either side and the table appears nonexistent rather than access-denied.

Test IAM in isolation before debugging Spark. Run aws glue get-table --database-name my_db --name my_table from a terminal with the same role credentials. If that returns the table metadata, IAM is not your problem.

Starting in 2025, AWS introduced the Glue Iceberg REST Catalog endpoint as part of Amazon SageMaker Lakehouse. This changes the integration pattern significantly. Instead of configuring GlueCatalog plus S3FileIO plus a warehouse path, you point Spark at a REST endpoint that handles catalog resolution, credential vending, and file I/O configuration server-side.

The configuration shrinks to:

`` spark.sql.catalog.glue_rest = org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.glue_rest.type = rest spark.sql.catalog.glue_rest.uri = https://glue.us-east-1.amazonaws.com/iceberg spark.sql.catalog.glue_rest.rest.sigv4-enabled = true spark.sql.catalog.glue_rest.rest.signing-region = us-east-1 spark.sql.catalog.glue_rest.rest.signing-name = glue ``

This eliminates the catalog-impl, io-impl, and warehouse properties entirely. The REST endpoint handles S3 credential vending through temporary STS tokens, so the cluster's IAM role only needs permission to call the Glue REST API — not direct S3 access to the data files. The endpoint returns pre-signed URLs or vended credentials per table operation.

The tradeoff: REST catalog support requires Iceberg 1.6.0 or later and Databricks Runtime 14.3+. Older clusters can't use it. And the REST endpoint is region-specific — if your Glue catalog is in us-west-2 but you point the URI to us-east-1, you'll get a valid HTTP response with zero tables.

For teams already on Unity Catalog, Databricks offers a parallel path: federate the Glue catalog as a foreign catalog in Unity Catalog. This lets you query Glue-registered Iceberg tables using standard three-part names (glue_federation.database.table) without any Spark catalog configuration at the cluster level. Unity Catalog handles the catalog resolution, and governance policies carry over. This requires Databricks Runtime 16.4 LTS or later and Unity Catalog with catalog federation enabled.

There's a subtler failure mode that no error message will catch. Iceberg tables use a metadata pointer — a metadata-location property in the Glue table definition — that points to the current metadata.json file in S3. When a writer (say, an EMR job or a Glue ETL job) commits new data, it updates this pointer atomically in Glue. But if your Databricks job caches the catalog metadata at session start, it may read a stale pointer for the duration of the session.

Spark's catalog caching behavior depends on the spark.sql.catalog.glue_catalog.cache-enabled property (defaults to true) and cache.expiration-interval-ms (defaults to 30 seconds in recent Iceberg versions). In a long-running Spark session — a streaming job or an interactive notebook left open for hours — the cache can serve outdated metadata indefinitely if the expiration is misconfigured or if the session pre-dates a schema evolution on the Iceberg table.

The result: your Databricks job completes successfully, returns rows, logs no errors — but the data is from a previous snapshot. Downstream dashboards show yesterday's numbers. A stakeholder flags the discrepancy at 9 AM. You check the job logs: everything passed. The pipeline monitoring tool shows green. The problem is invisible to any system that only checks job exit codes.

This is where observability beyond exit-code monitoring matters. MetricSign tracks Databricks job outputs against expected freshness windows and flags refresh_delayed when the data a job produces hasn't changed within the expected interval — even when the job itself reports success. For Iceberg-based pipelines where stale metadata reads produce silently correct-looking results, that freshness signal is the difference between catching the issue at ingestion time and hearing about it from the CFO.

To prevent stale reads operationally, set cache.expiration-interval-ms to a value shorter than your pipeline's scheduling interval, or disable caching entirely for batch jobs with spark.sql.catalog.glue_catalog.cache-enabled=false. For streaming jobs, use Iceberg's refresh() method to force a metadata reload before each micro-batch.

Debugging Iceberg-Glue-Databricks integration is frustrating because the failure modes are indistinguishable from each other at the Spark SQL level. Here's a systematic approach that eliminates variables one at a time.

First, verify the JAR is on the classpath. Run spark.sparkContext.getConf.get("spark.jars.packages") or check the cluster's Libraries tab. The JAR must match your Spark major.minor version exactly. For Spark 3.5 on Scala 2.12, that's org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.0 or later.

Second, verify the catalog registered. Run spark.catalog.listCatalogs() in a Python cell or SHOW CATALOGS in SQL. If glue_catalog (or whatever name you chose) isn't listed, one of the spark.sql.catalog.* properties has a typo or the JAR failed to load.

Third, verify Glue connectivity independently. Run aws glue get-databases using the same IAM role as the cluster. If this returns databases, Glue permissions are fine. If not, fix IAM before touching Spark.

Fourth, verify the warehouse path format. It must be a full S3 URI with a trailing slash: s3://bucket/path/. Some Iceberg versions also accept s3a://, but mixing s3:// and s3a:// between the warehouse property and the table's actual data location causes path resolution failures.

Fifth, check Spark driver logs for the actual exception. Notebook output truncates stack traces. The driver's stderr log contains the full GlueException, S3Exception, or InstantiationException that Spark wraps in AnalysisException. On Databricks, access these through the Cluster page → Driver Logs tab, or configure log delivery to S3/CloudWatch for persistent access.

Finally, if you're on Databricks Runtime 14.3+ and using Iceberg 1.6+, consider switching to the REST catalog endpoint or Unity Catalog federation. Both reduce the configuration surface dramatically and eliminate the most common misconfiguration vectors.