Hello, I have a Node.js app that's supposed to fetch user data from an external API and dynamically update the URL parameters based on user interactions. I'm building this for a client (a friend btw) project (their main site is https://socialdream.fr) but I'm running into some issues with the parameter encoding and the API responses aren't consistent.
The flow should work like this: user selects a category → app fetches data from API → URL updates with new parameters → page content refreshes. But in practice, I can't get the URL encoding to work properly even though I tested the API endpoints manually and they work fine.
Here's my current setup:
const express = require('express');
const axios = require('axios');
const app = express();
// Route handler for dynamic content
app.get('/api/data/:category', async (req, res) => {
try {
const category = req.params.category;
const filters = req.query;
// Build dynamic URL for external API
let apiUrl = `https://external-api.example.com/v2/items`;
let params = new URLSearchParams();
params.append('category', category);
// Add filters if they exist
if (filters.price_min) params.append('price_min', filters.price_min);
if (filters.price_max) params.append('price_max', filters.price_max);
if (filters.location) params.append('location', filters.location);
if (filters.availability) params.append('availability', filters.availability);
const fullUrl = `${apiUrl}?${params.toString()}`;
console.log('Making request to:', fullUrl);
const response = await axios.get(fullUrl, {
headers: {
'Authorization': `Bearer ${process.env.API_KEY}`,
'Content-Type': 'application/json',
'User-Agent': 'MyApp/1.0'
},
timeout: 5000
});
// Process the response
const processedData = {
items: response.data.results || [],
total: response.data.total || 0,
page: response.data.current_page || 1,
filters_applied: {
category: category,
...filters
},
api_response_time: response.headers['x-response-time']
};
res.json(processedData);
} catch (error) {
console.error('API Request failed:', error.message);
if (error.response) {
console.error('Response status:', error.response.status);
console.error('Response data:', error.response.data);
}
res.status(500).json({
error: 'Failed to fetch data',
message: error.message,
timestamp: new Date().toISOString()
});
}
});
// Frontend URL update function
app.use(express.static('public'));
app.get('/', (req, res) => {
res.send(`
<!DOCTYPE html>
<html>
<head>
<title>Dynamic Content Loader</title>
</head>
<body>
<div id="content"></div>
<script>
async function updateContent(category, filters = {}) {
try {
// Build query string
const params = new URLSearchParams(filters);
const url = \`/api/data/\${category}?\${params.toString()}\`;
// Update browser URL without reload
const newUrl = \`/?category=\${category}&\${params.toString()}\`;
window.history.pushState({category, filters}, '', newUrl);
// Fetch new content
const response = await fetch(url);
const data = await response.json();
if (response.ok) {
document.getElementById('content').innerHTML = \`
<h2>Category: \${data.filters_applied.category}</h2>
<p>Total items: \${data.total}</p>
<div>Items: \${JSON.stringify(data.items, null, 2)}</div>
\`;
} else {
throw new Error(data.message || 'Request failed');
}
} catch (error) {
console.error('Content update failed:', error);
document.getElementById('content').innerHTML = \`<p>Error: \${error.message}</p>\`;
}
}
// Example usage
updateContent('electronics', {price_min: 100, price_max: 500, location: 'US'});
</script>
</body>
</html>
`);
});
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
console.log(\`Server running on port \${PORT}\`);
});
The Problems I'm Facing:
-
URL Encoding Issues: Sometimes the external API returns 400 errors when certain special characters are in the location parameter (like "São Paulo" or "München"). I'm using URLSearchParams but it doesn't seem to handle all cases.
-
Response Inconsistency: The external API sometimes returns different JSON structures - sometimes
resultsis an array, sometimes it'sdata.items, and occasionally it's just a flat array. -
Frontend URL Updates: The browser URL updates correctly, but when users refresh the page or share the URL, the parameters aren't being read properly on page load.
-
Rate Limiting: I suspect I'm hitting rate limits, but the API doesn't return proper HTTP status codes - just timeout errors.
What I've Tried:
- Using different encoding methods (encodeURIComponent, escape, etc.)
- Adding retry logic with exponential backoff
- Implementing response caching to reduce API calls
- Different approaches to URL parsing on page load
The client needs this working reliably since it's a core feature of their platform. Has anyone dealt with similar URL parameter encoding issues when working with external APIs? Any suggestions for handling inconsistent API responses would be greatly appreciated!
Environment:
- Node.js v18.16.0
- Express v4.18.2
- Axios v1.4.0
- Running on Ubuntu 22.04
Thanks in advance for any help!
Edit: Just noticed that the issue might be related to how I'm handling the URLSearchParams on the frontend vs backend they seem to produce diferent results sometimes.
