[SIP-61] Proposal for improving the organization of front-end folders

Motivation

The major motivation of this PR is to propose a better organization for our front-end folders.

This is our current structure:

branding (old files)
cypress-base (e2e tests)
images (images)
spec (unit and integration tests)
stylesheets (less files)
src
   addSlice (page)
   api (one service)
   chart (components)
   common (common components and hooks)
   components (common components)
   CRUD (components)
   dashboard (page)
   dataMask (action, reducer and type)
   datasource (components)
   explore (page)
   filters (components)
   logger (utilities)
   messageToasts (components)
   middleware (initialization)
   modules (utilities)
   profile (page)
   setup (initialization)
   showSavedQuery (utilities)
   SqlLab (page)
   staticPages (assets)
   types (generic types)
   utils (utilities)
   views (multiple pages)
      CRUD
         alert (page)
         annotation (page)
         annotationlayers (page)
         chart (page)
         csstemplates (page)
         dashboard (page)
         data 
            database (page)
            dataset (page)
            query (page)
            saved query (page)
         welcome (page)
   visualizations (components)

As you can see I tagged each folder with its respective role in the application. This structure has some problems:

• Lack of organizational consistency. We don’t know if the folders are organized by features, pages, technical layers, modules, etc. They are all mixed.
• Lack of naming consistency. Some folder uses camel case pattern while others use snake case (SqlLab, addSlice, annotationlayers).
• Lack of test coverage. We have separate folders for tests and components (src e spec). The spec folder tries to mimic the same structure of the src folder but has a lot of folders and components missing. This indirect relationship contributes to decreasing test coverage.
• Lack of documentation about the structure and its justifications. New contributors don't know where to develop new features or have difficulty finding a particular feature.

With that in mind, this PR tries to achieve the following:

• Create an organizational structure based on common best practices but tailored to our project
• Create a document about the organizational structure for the community
• Remove conceptual conflicts
• Fix folder naming inconsistencies
• Create an organizational structure that encourages testing and reduces coverage problems
• Guide new features

Proposed Change

The proposed change is based on a number of resources available. Here are some of them:

• A feature-based approach to React development
• React Folder Structure in 5 Steps
• How to better organize your React applications?
• Why React developers should modularize their applications?
• React Project Structure Best Practices for Scalable Application
• React JS— Architecture + Features + Folder structure + Design Pattern

I took the liberty to paste some quotes from the sources:

Organizing your code by modules means you start thinking around a concept where you break down your code into related pieces and provide a public interface to use your module. It helps maximize code sharing and reusability in different sections of your application and even on other projects.

Another important attribute of a module-based structure is the ability to replace one by another, as long as both modules meet the same requirements.

In a world where everything moves fast and new features replace old ones in no time, breaking your code by modules will give you the freedom to build multiple versions of them to facilitate A/B testing and fully replace old ones when your team is satisfied with the result.

Each component, scene, or service (a feature) has everything it needs to work on its own, such as its own styles, images, translations, set of actions as well as a unit or integration tests. You can see a feature like an independent piece of code you will use in your app (a bit like node modules).

Code is structured in a way that reflects purpose.

Code is easier to refactor.

Organizing an application as a series of features can make feature flags easier to implement.

The idea is to better organize our code structure to reflect a more modular approach. Applying those concepts to our code base is an iterative process by nature, so the following proposed structure is the result of the first iteration:

cypress-base (e2e tests)
src
   assets (common assets)
      images (images)
      staticPages (assets)
      stylesheets (less files)		
   components (common components)
      chart (components)
      dataMask (action, reducer and type)
      datasource (components)
         CRUD (components) - this folder won’t exist. It’s here just to show where the components went.
      messageToasts (components)
      visualizations (components)
         filters (components)
   hooks (common hooks)
   pages
      addSlice (page)
      alert (page)
      annotation (page)
      annotationLayers (page)
      chart (page)
      cssTemplates (page)
      dashboard (page)
      database (page)
      dataset (page)
      explore (page)
      profile (page)
      query (page)
      savedQuery (page)
      sqlLab (page)
         api (one service)
      welcome (page)
   setup (initialization)
      middleware (initialization)
   types (common types)
   utils (utilities)
      logger (utilities)
      modules (utilities)
      showSavedQuery (utilities)

These are the changes:

• All folders with camel case naming.
• The majority of articles put the assets inside src, so we can move images, stylesheets, and staticPages to a folder named asset under src.
• Only one image of the branding folder is referenced and the same image is present in the images folder. So we can remove the branding folder and update the reference.
• All components that are shared across the application are inside the components folder. A component that is specific for a page should go inside that page.
• All common hooks are inside the hooks folder.
• The pages folder is the heart of the application. All the various features/screens/pages are defined here. Each page will have a components dir. This will hold all the components that are required by only this page.
• The setup folder is responsible for the initialization code.
• The types folder are common types used throughout the application. When possible we should define types in the same files of their components.
• The utils folder contains all the utilities.
• All the specs were moved to the src folder alongside each component/page/hook/etc. The same is valid for mocks. When restructuring we should verify if a mock is shared or belong to a specific module.
• I opted for leaving e2e tests outside of src because it seems to be the common practice and because these tests are designed to span across multiple pages and workflows.

Looking at a more detailed view of the structure, we should strive to organize our code in a more modular way. Some examples here for demonstration purposes:

An example of a common component with storybook and tests:

src/components/Button
   Button.test.tsx
   Button.stories.tsx
   index.tsx

An example of a page:

src/pages/sqlLab
   actions
      sqlLab.ts
      sqlLabl_spec.tsx
   components
      QueryTable
         index.tsx
         QueryTable.test.tsx 
      ResultSet
         index.tsx
         ResultSet_spec.tsx
   reducers
      sqlLab.ts
      sqlLab.test.tsx
   utils 

Here we can see that we can create nested folders when necessary to better organize our code. Also, the most important aspect here is that the tests are close to the component and this will help to spot coverage problems in our application.

In the future we can iterate more on this structure until we reach something like this:

src/pages/sqlLab
   FeatureA
      actions
      components
      reducers
   FeatureB
      actions
      components
      reducers

When we reach this point, all related code will be grouped together and it will be even easier to work with feature flags and A/B feature substitutions.

The proposed structure is not final. It is the result of the first iteration following the organizational principles described here. As we move forward in the organization of the folders, we can make different decisions motivated by technical restrictions or a better understanding of a component. The idea is to start the code organization by taking an iterative approach and construct a guideline in which all organizational decisions are justified and can be easily understood by the community.

New or Changed Public Interfaces

The intent of this PR is to not change any public interface.

New dependencies

No new dependencies.

Migration Plan and Compatibility

The idea here is to promote this organization through small PRs, using an iterative approach, to avoid abrupt changes and reduce conflicts with open PRs. Similar to what we did in the components folder. After every step, we should assess and review the structure.

We should also create from the start the documentation about the structure. We can write a section in CONTRIBUTING.MD or, because of CONTRIBUTING.MD size, another file like FRONTEND_STRUCTURE.MD. This documentation will guide the efforts and serve as a reference for the community.

Rejected Alternatives

Another popular way to structure projects is to group similar files together, by file type or technical layers. This could be applied to small to medium projects but doesn’t scale as well as grouping by features.

Keep the structure as it is was also rejected because of all the problems listed above.