Why Databricks Notebooks Reject raw_input — and How to Authenticate APIs Without a Terminal

The error is blunt:

StdinNotImplementedError: raw_input was called, but this frontend does not support input requests.

This surfaces when a Python library calls input() (or raw_input() in Python 2) to prompt the user for something — a URL to paste back, a confirmation, a code. In a local terminal, that works. In a Databricks notebook, there is no stdin file descriptor attached to the kernel process. The IPython kernel that powers Databricks notebooks explicitly raises StdinNotImplementedError instead of hanging forever on a read that will never complete.

The Spotify API is a common trigger, but it is not the only one. Any OAuth 2.0 Authorization Code flow that uses a local redirect and then asks the user to paste a URL back will hit this wall. The spotipy library's SpotifyOAuth.get_access_token() method, when it cannot open a browser and has no cached token, falls through to an input() call asking for the redirect URL. That single line kills the notebook cell.

The important thing to understand: this is not a bug in spotipy or in Databricks. It is an architectural mismatch. Authorization Code flow was designed for applications with a user sitting at a browser. Databricks notebooks are headless compute environments that happen to have a web UI for editing code. The notebook UI does not proxy stdin to the kernel. No amount of configuration will change that — you need a different authentication strategy.

If your workload does not need user-specific data — you are pulling public playlists, track metadata, audio features, or catalog search results — switch from SpotifyOAuth to SpotifyClientCredentials. This is the single most common fix for this error.

import spotipy
from spotipy.oauth2 import SpotifyClientCredentials

auth_manager = SpotifyClientCredentials(
    client_id=dbutils.secrets.get(scope="spotify", key="client_id"),
    client_secret=dbutils.secrets.get(scope="spotify", key="client_secret")
)
sp = spotipy.Spotify(auth_manager=auth_manager)

# This works — no input() call, no browser redirect
results = sp.search(q="artist:Radiohead", type="track", limit=10)

Client Credentials flow is a direct POST to https://accounts.spotify.com/api/token with your client ID and secret. Spotify returns an access token valid for 3600 seconds. No user consent screen, no redirect URI, no stdin. The spotipy library handles refresh automatically.

Two things to note. First, store your credentials in Databricks secrets (dbutils.secrets), not in notebook cells. Hardcoded secrets in notebooks get committed to version control, logged in job output, and visible in the Databricks workspace to anyone with read access. Second, Client Credentials tokens cannot access user-private endpoints: saved tracks, playlists marked private, user profile details, playback control. If you need those, read the next section.

When you genuinely need user-scoped access — pulling a user's saved library, reading private playlists, or writing to a playlist on their behalf — you need an Authorization Code token. But you cannot run that flow inside Databricks. The solution is to separate token acquisition from token usage.

Step 1: Run the OAuth flow locally. On your laptop, execute the authorization once:

from spotipy.oauth2 import SpotifyOAuth

sp_oauth = SpotifyOAuth(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    redirect_uri="http://localhost:8080/callback",
    scope="user-library-read playlist-read-private",
    cache_path=".spotify_token_cache"
)
token_info = sp_oauth.get_access_token(as_dict=True)

This opens your browser, you consent, and spotipy writes the token (including the refresh token) to .spotify_token_cache.

Step 2: Store the refresh token in Databricks secrets:

databricks secrets put-secret spotify refresh_token --string-value "AQD..."

Step 3: In your Databricks notebook, build a token refresh mechanism that never calls input():

import requests

def refresh_spotify_token():
    refresh_token = dbutils.secrets.get(scope="spotify", key="refresh_token")
    client_id = dbutils.secrets.get(scope="spotify", key="client_id")
    client_secret = dbutils.secrets.get(scope="spotify", key="client_secret")
    response = requests.post(
        "https://accounts.spotify.com/api/token",
        data={
            "grant_type": "refresh_token",
            "refresh_token": refresh_token,
            "client_id": client_id,
            "client_secret": client_secret
        }
    )
    return response.json()["access_token"]

sp = spotipy.Spotify(auth=refresh_spotify_token())

This pattern works for any OAuth provider, not just Spotify. The localhost redirect URI that the original community poster used is fine for the local step — it just cannot run inside Databricks.

Spotipy's architecture has a cleaner escape hatch than raw requests.post calls: the CacheHandler interface. Instead of letting spotipy try to read and write .cache files (which also causes permission issues on DBFS), inject a MemoryCacheHandler pre-loaded with your token info.

from spotipy.oauth2 import SpotifyOAuth
from spotipy.cache_handler import MemoryCacheHandler

token_info = {
    "access_token": "current_access_token_here",
    "refresh_token": dbutils.secrets.get(scope="spotify", key="refresh_token"),
    "token_type": "Bearer",
    "expires_in": 3600,
    "scope": "user-library-read playlist-read-private",
    "expires_at": 0  # Forces immediate refresh
}

cache_handler = MemoryCacheHandler(token_info=token_info)
auth_manager = SpotifyOAuth(
    client_id=dbutils.secrets.get(scope="spotify", key="client_id"),
    client_secret=dbutils.secrets.get(scope="spotify", key="client_secret"),
    redirect_uri="http://localhost:8080/callback",
    scope="user-library-read playlist-read-private",
    cache_handler=cache_handler,
    open_browser=False
)

sp = spotipy.Spotify(auth_manager=auth_manager)

Setting expires_at to 0 forces spotipy to refresh the token using the refresh token on the first API call. Because the refresh grant is a server-to-server POST, it never triggers input(). The open_browser=False flag prevents the fallback browser-open attempt, which also fails silently in Databricks since there is no desktop environment.

This approach keeps you within spotipy's managed token lifecycle — automatic refresh, scope validation, retry logic — without the StdinNotImplementedError. It also avoids the file system issues that CacheFileHandler causes on cluster-attached storage, where the working directory may reset between job runs or across cluster restarts.

The community thread ended without a confirmed resolution, which is typical. Someone gets the notebook working, moves on, and schedules it as a Databricks job. Then it fails silently three weeks later.

Here is why. Spotify refresh tokens issued under the Authorization Code flow do not expire on a fixed schedule, but they can be revoked: when the user changes their password, when the app's API credentials are rotated, or when Spotify invalidates tokens during security events. When that happens, the refresh call returns HTTP 400 with {"error": "invalid_grant"}. Your job fails. If you are not monitoring job outcomes, the data just stops updating.

For Databricks jobs pulling from external APIs, the failure mode is not an immediate crash — it is stale data. The job runs, the token refresh fails, the API call returns 401, and depending on your error handling the job might even succeed with zero rows written. Downstream tables look current because the job timestamp updates. The data does not.

MetricSign monitors Databricks job outcomes and flags runs where output volume drops to zero or deviates from historical patterns. Instead of discovering three weeks later that your Spotify pipeline silently stopped ingesting data, you get an alert on the first anomalous run with context about what changed.

The broader lesson applies beyond Spotify. Any Databricks job that depends on external OAuth tokens — Salesforce, HubSpot, Google Analytics, Stripe — carries the same risk. Build token validation into your pipeline: check the HTTP status of a lightweight API call before running the full extraction, and raise an explicit exception on 401. Do not let a silent auth failure masquerade as an empty result set.

Spotipy is the most-discussed case, but any Python package that calls input() at runtime will fail identically in Databricks. Common culprits include:

The google-auth-oauthlib library's InstalledAppFlow.run_console() method, which prompts the user to visit a URL and paste back an authorization code. In Databricks, use google.oauth2.service_account.Credentials.from_service_account_info() instead, loading the service account JSON from Databricks secrets.

The tweepy library's OAuth 1.0a flow (OAuthHandler.get_authorization_url() followed by input() for the verifier PIN). Use Bearer Token authentication (tweepy.Client with a bearer token) for read-only access, or pre-authorize and store tokens.

Any custom script that uses input() for confirmation prompts like input('Are you sure you want to delete? y/n'). Replace these with explicit boolean parameters or configuration flags when running in automated environments.

The fix is always the same structural change: separate the interactive human step from the automated compute step. Authenticate where you have a browser and a keyboard. Execute where you have compute and a scheduler. Pass tokens — never prompts — between the two.

You can detect this proactively by searching your codebase for input( calls in any module that will run on Databricks:

import inspect, spotipy
source = inspect.getsource(spotipy.oauth2)
assert 'input(' not in source, 'This module uses input() — will fail in Databricks'

That assertion is crude but effective as a pre-flight check in your notebook's setup cell.