[FIXED] Persistent Connections in Django?

Issue

I am trying to understand how persistent connections improve performance especially when my application is READ heavy. So, I have a record of 100000 rows and I have indexed the username field. So, I have created an endpoint where I can query using a username and it gives me the temperature. This is just for testing purposes. So, I wrote another script that has a function that creates a loop that runs 1000 times and it makes a request to the endpoint. I was logging the request-response time and saving it in a list and then calculating the total time taken to complete the 1000 requests. First, I tried with the default Django database settings which don't have persistent connections, and every time I benchmarked the time and it logs around 43 seconds for all 1000 requests. Next, I added CONN_MAX_AGE=60 and ran the script multiple times, found out the time increases to 103 seconds every time. Now, if I am not wrong, a persistent connection should have reduced the total time but for some reason, it was way higher than the normal connections which open and close every time a request-response is done. My question is why is this happening? script.py

import requests
import random
import time
def make_request():
    username = ['00b076300079144ca0667c341aeac046','remy','surajit','remy11']
    t = []

    for i in range(1000):

        response = requests.get('http://127.0.0.1:8082/temperature/t/{}/'.format(random.choice(username)))
        t.append(response.elapsed.total_seconds())
        print(random.choice(username),response.status_code,response.elapsed.total_seconds())
    print('total time',sum(t))

make_request()

DetailView endpoint

class TempratureDetailView(DetailView):
    model = Temperature
    template_name = 'temperature/t_detail.html'

    def get_object(self, queryset=None):
        return Temperature.objects.get(username=self.kwargs.get('username'))

models.py

class Temperature(models.Model):
    username = models.TextField(db_index=True,primary_key=True)
    temperature = models.IntegerField(blank=True, null=True)

    class Meta:
        db_table = 'temperature'

Persistent database settings

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.postgresql_psycopg2',

        'NAME': '',

        'USER': '',

        'PASSWORD': '',

        'HOST': 'localhost',

        'PORT': '',

        'CONN_MAX_AGE': 600, #10 minutes

    }
}

Solution

From the clarification in your comment, you are running these tests on a development server.
Quoting Django's documentation:

The development server creates a new thread for each request it handles, negating the effect of persistent connections. Don’t enable them during development.

Basically each thread starts a connection of it's own to the database, considering the number of requests made here in the tests, and the setting of CONN_MAX_AGE=60, most likely your database doesn't support as much connections as the server is making in this situation, causing longer waits for previous connections to be closed, and also negating the usage of persistent connections. A better test would need to be done on a production server given the circumstances to arrive on a proper conclusion.

Answered By - Abdul Aziz Barkat