"When I started this project, I had no clear concept of the time and money it would take. After rewriting the app multiple times, I believe it's crucial to reflect on the journey and pinpoint essential steps in constructing a data library. In my retrospective, I'll share insights on maintaining a sustainable data workflow, and I'm excited to hear your feedback."
It requires time commitment
My biggest investment was time. In mid-2022, I dedicated an entire month to researching and consolidating resources on web and database development. This involved reading documentation, following tutorials, and building an AJAX data management system, along with data visualizations in JavaScript, which took me a few more months.
The whole process, from defining objectives and key results to the first successful deployment, took five months. Another two months were spent finalizing the content management system setup and interface design. I also consider the time spent on activities like hiking, as it allowed me to unconsciously digest knowledge and find solutions while not actively coding.
I faced challenges and setbacks in every step, leading to additional delays in the timeline. However, I see this as a normal part of the learning process. The key is to learn from both successes and mistakes, ultimately building an infrastructure that can be reused for future expansion.
A cozy setup to work more efficiently :)
- Two 27-inch monitors
- Mechanical keyboards with hand rests
- Ergonomic mouse
- Noise-canceling headphones
- Ergonomic chair with back and arm supports
The data should be legitimate and well-maintained
Maintaining a robust data life cycle demands careful attention, logic, and quality assurance at every stage of acquisition, examination, storage, and update.
- Data ethics: it's essential to note that data is non-material and should be sourced from publicly accessible channels such as company websites, government sites, news releases, and annual reports.
- Data cleaning, modeling, and Proof of Concept (POC): break down initial data into reusable parts and storing them separately. A well-designed model maximizes flexibility for future coverage expansion.
- Prototyping: align closely with clearly defined business needs for effective development.
An innovative data management system infused with data science
At the heart of any research lies data, and effective data management is the cornerstone of a successful business. A data scientist or researcher is rendered ineffective without access to high-quality data.
Whether stored on a local PC or expanded to a web server for collaborative team efforts, the key lies in ensuring continuous system updates aligned with market trends and evolving business requirements. Eventually, there comes a point where a comprehensive rebuild becomes necessary.
Considerations include:
- Selecting the right database, tech stack, and framework.
- Defining procedures: Implementing Business Rule Engine (BRE) in the frontend, backend, and database. Understanding the unique characteristics and requirements of each dataset.
- Data visualization and analysis: Beyond mere exploration and market research, internal analysis may delve into data quality.
Feedbacks matters
It's crucial to gather diverse perspectives, now that the website is online:
- Industry experts: These professionals are not just potential customers; they provide valuable, professional advice. During demos, the questions they pose become benchmarks, testing product coverage and shaping future goals.
- Tech Department Guidance: Insights from the tech department establish a contemporary standard for designing data products. This includes considerations like data connections, API usage, and pricing strategies.
- Fresh Perspectives: Seeking advice from individuals unfamiliar with the topic assesses the effectiveness of information conveyed to visitors and the logic of website navigation.
I hope this helps when you’re creating your data product, whether it’s for a company or just a personal passion like mine. And most importantly, I hope you have a lot of fun doing it!