Best way to create and utilize python project structure

Over the past few years python language has been steadily growing in popularity and its use in data science and automation has been on the rise. Before deep diving into the development of python application, let's understand the project structure.

Why project structure is important?

Will take a simple example. Let's say we need to create an application for one of the colleges where all college-related functionality should be present like students and their exams attendance, teachers and their subjects etc.

Nowadays to do rapid development everyone is preferring to have microservice architecture. In our case also will create separate projects or modules for students, teachers, exams etc. and each developer will work on their modules. But what if the developer working on students project wants to use the functionality of the exams and teachers projects? That's where project structure plays an important role.

Following are a few more use cases

• Consume auth functionality for different types of auth mechanisms.

• Creating helper utilises like date parsing, formatting, email sending etc.

• Exposing rest endpoints using Flask by just importing projects.

What should be the project structure?

project_name/__init__.py
project_name/models/__init__.py
project_name/services/__init__.py
project_name/controllers/__init__.py
project_name/constants/__init__.py
project_name/configuration/env.ini
setup.py
requirements.txt
README.md

• project_name package: Always create a single package with the name same as your project name. Rest all packages will be defined under this package eg. models, services etc. The reason behind this is to avoid discrepancies in installed site-packages. Let's say you haven't created project_name package under students and teachers projects and you directly created the package name services. Now consider teachers project want to utilize students services then you will install students project with pip install <student_project_path> and try to import services.attendance but in that case python will prioritize the relative path package which is under teachers projects. This will create a discrepancy.

• models package: This is optional. You can add your data models such as student.py, teacher.py.

• services package: Where actual business logic is placed. eg services/attendance.py module will have an attendance logic.

• controllers package: If you are going to expose rest endpoints then you can create this package. eg. students_controller.py contains @route("/list") route

• constants package: whenever you want to create a singleton object you better define inside constants.__init__.py and then you can import anywhere.

• configuration: Where you define python supported environment specific configuration file. You can provide file path as project_name/configuration/env.ini to set your configuration.

• setup.py: Used to build and distribute Python packages. You can install your project as a python package if you have written setup.py properly. Ref link

• requirements.txt: List down all dependent packages, modules and libraries which you have used in your project. Useful to create your virtual environment to run your application.

• README.md: You can mention the description of your project and how to utilize it.

Example

Le's take the above example and create out project structure for students and teachers.

setup.py for both projects

• students setup.py

    from setuptools import setup
    setup(
         name="students",
         version="0.1",
         description="Students functionality",
         packages = [ "students", "students.models", "students.services", "students.controllers", "students.constants" ],
         install_requires=[]
    )
  

• teachers setup.py

    from setuptools import setup
    setup(
        name="teachers",
        version="0.1",
        description="Teachers functionality",
        packages = [ "teachers", "teachers.models", "teachers.services", "teachers.controllers", "teachers.constants" ],
        install_requires=[]
    )
  

Description of above attributes

• name: Name of the python package.

• version: You can have multiple versions for each feature you are building it. So whoever wants a specific version can use it.

• description: Define a short description of your application.

• packages: List down the packages which you want to expose.

• install_requires: List down dependent packages or modules required to run your application. Whenever you install your application as a package, your install_requires packages will get installed automatically.

How to utilize it?

Consider the following code snippet present in teachers service and students service respectively

from students.services.service import get_attendance
def get_marks_for_attendance(student_id):
    attendance_percentage = get_attendance(student_id)
    if attendance_percentage > 75:
        return 15
    return 0

def get_attendance(student_id):
    # get it from db
    return 80

But how can we use get_attendance function inside teachers service? There are two ways.

1. Install from local file path.

   • eg. pip install <students_project_path>

2. Install from devpi server Ref link for devpi basics and setup.

   • After setting up devpi you can upload students and teachers project on devpi server

   • After uploading it you can install via devpi.

     • eg.pip install -i localhost:3141<user>/<index> students==0.1, pip install -i localhost:3141<user>/<index> teachers==0.1

Conclusion

Projects like small integration with a third party, data science project, and microservice architectural projects use python language for easy development. To make the process faster and reutilize your own or your team member's functionality you will follow described project structure and utilize it.