High-Performance IoT Monitoring With Redis-Backed In-Memory Storage and Containerized CI/CD Pipeline

Abstract

This study presents the design, development, and deployment of a high-performance Internet of Things (IoT) monitoring system featuring an integrated Redis-based in-memory processing architecture. The system collects real-time environmental data—including temperature, humidity, and pressure—from field-deployed TI CC1352R SensorTag devices operating within an LR-WPAN network. These measurements are forwarded to a TI CC1352R LaunchPad border router node, which employs an ESP32 Wi-Fi SoC interface to transmit the sensed data to cloud-based web services via the global Internet. The web service dynamically caches and stores incoming traffic according to database write speed and network intensity, while the Redis caching layer significantly enhances data ingestion and monitoring response times by reducing database load. The backend follows modular and scalable CI/CD pipeline principles, automating the testing, building, and deployment phases. Except for the firmware running on sensor nodes, every component of the system is triggered by version-controlled merges, ensuring automated testing and re-deployment. One of the main challenges in complex IoT systems—especially those requiring continuous weekly feature updates (sprint-based development)—is constructing a fully integrated, end-to-end system that remains stable and extensible. Experimental findings demonstrate rapid recovery from fault conditions (with total re-deployment times below 10 minutes), efficient real-time data visualization, and improved system resilience. The proposed architecture also provides practical guidance for researchers focusing on IoT systems that traditionally rely on simulations by offering a fully operational, end-to-end technological solution. By caching large volumes of sensor data before disk storage, the system overcomes one of the major scalability bottlenecks in growing IoT networks. On a 12 GB RAM server, the architecture successfully stores data from approximately 530,000 sensor messages using a 10-second caching interval, effectively absorbing sudden data bursts through Redis buffering. Consequently, the study establishes a robust foundation for future extensions involving machine learning integration and large-scale sensor deployments.

Keywords

• Internet of things (IoT)
• In-memory processing
• Redis
• Containerization
• CI/CD pipeline

References

1. Akkaş, M. A., Sokullu, R., & Çetin, H. E. (2020). Healthcare and patient monitoring using IoT. Internet of Things, 11, 100–173.
2. Alexander, M. (2008). Survey, surveillance, monitoring and recording. In W. J. Sutherland, L. V. Dicks, N. Ockendon, & R. K. Smith (Eds.), Management planning for nature conservation: A theoretical basis & practical guide (pp. 49–62). English Nature.
3. Alfian, G., Syafrudin, M., & Rhee, J. (2017). Real-time monitoring system using smartphone-based sensors and NoSQL database for perishable supply chain. Sustainability, 9(11), 2073.
4. Amghar, S., Cherdal, S., & Mouline, S. (2018). Which NoSQL database for IoT applications? In 2018 International Conference on Selected Topics in Mobile and Wireless Networking (MoWNet) (pp. 131–137). IEEE.
5. Cambra, C., Sendra, S., Lloret, J., & Garcia, L. (2017). An IoT service-oriented system for agriculture monitoring. In IEEE International Conference on Communications (ICC) (pp. 1–6). IEEE.
6. Chauhan, V., Patel, M., Tanwar, S., Tyagi, S., & Kumar, N. (2020). IoT enabled real-time urban transport management system. Computers & Electrical Engineering, 86, 106746.
7. ClickUp. (2025). ClickUp: One app to replace them all. https://clickup.com/ (Accessed March 27, 2025)
8. Cloudinary. (2025). Cloudinary Ltd – Media management platform. https://cloudinary.com (Accessed March 23, 2025)

9. Costantino, D., Malagnini, G., Carrera, F., Rizzardi, A., Boccadoro, P., Sicari, S., & Grieco, L. A. (2018). Solving interoperability within the smart building: A real test-bed. In IEEE International Conference on Communications Workshops (ICC Workshops) (pp. 1–6). IEEE.
10. Darshan, K., & Anandakumar, K. (2015). A comprehensive review on usage of Internet of Things (IoT) in healthcare system. In International Conference on Emerging Research in Electronics, Computer Science and Technology (ICERECT) (pp. 132–136). IEEE.
11. Docker. (2020). Docker: Empowering app development for developers. https://www.docker.com (Accessed March 27, 2025)
12. Docker. (2025). Docker Compose. https://docs.docker.com/compose/ (Accessed March 27, 2025)
13. Dunkels, A., Gronvall, B., & Voigt, T. (2004). Contiki: A lightweight and flexible operating system for tiny networked sensors. In 29th Annual IEEE International Conference on Local Computer Networks (LCN) (pp. 455–462). IEEE.
14. Espressif Systems. (2025). ESP32 series datasheet. https://www.espressif.com/en/products/socs/esp32/resources (Accessed March 27, 2025)
15. Fielding, R. T. (2000). Architectural styles and the design of network-based software architectures (Doctoral dissertation, University of California, Irvine).
16. Ghazivakili, M. (2021). Industry 4.0: Industrial IoT enhancement and WSN performance analysis (Doctoral dissertation, Politecnico di Torino).
17. GitHub. (2025a). GitHub Actions: Automate your workflow from idea to production. https://github.com/features/actions (Accessed March 29, 2025)
18. GitHub. (2025b). GitHub: Where the world builds software. https://github.com/ (Accessed February 27, 2025)
19. Glasgow, H. B., Burkholder, J. M., Reed, R. E., Lewitus, A. J., & Kleinman, J. E. (2004). Real-time remote monitoring of water quality. Journal of Experimental Marine Biology and Ecology, 300, 409–448.
20. Grafana Labs. (2025). Grafana: The open-source observability platform. https://grafana.com/ (Accessed January 18, 2025)
21. Hassija, V., Chamola, V., Saxena, V., Jain, D., Goyal, P., & Sikdar, B. (2019). A survey on IoT security. IEEE Access, 7, 82721–82743.
22. Hossein Motlagh, N., Mohammadrezaei, M., Hunt, J., & Zakeri, B. (2020). Internet of Things (IoT) and the energy sector. Energies, 13(2), 494.
23. IEEE. (2020). IEEE standard for low-rate wireless networks (IEEE Std 802.15.4-2020). https://standards.ieee.org/ieee/802.15.4/7029/ (Accessed January 27, 2025)
24. IEEE. (2021). IEEE standard for wireless LAN MAC and PHY specifications (IEEE Std 802.11-2020). https://standards.ieee.org/ieee/802.11_Edition/10098/ (Accessed January 27, 2025)
25. Jeong, S., Zhang, Y., O’Connor, S., Lynch, J. P., Sohn, H., & Law, K. H. (2016). A NoSQL data management infrastructure for bridge monitoring. Smart Structures and Systems, 17(4), 669–690.
26. Kang, Y. S., Park, I. H., Rhee, J., & Lee, Y. H. (2015). MongoDB-based repository design for IoT-generated RFID/sensor big data. IEEE Sensors Journal, 16(2), 485–497.
27. Krawczyk, H., Bellare, M., & Canetti, R. (1997). HMAC: Keyed-hashing for message authentication (RFC 2104). https://datatracker.ietf.org/doc/html/rfc2104 (Accessed April 15, 2025)
28. Li, T., Liu, Y., Tian, Y., Shen, S., & Mao, W. (2012). A storage solution for massive IoT data based on NoSQL. In IEEE International Conference on Green Computing and Communications (GreenCom) (pp. 50–57). IEEE.
29. Lovett, G. M., Burns, D. A., Driscoll, C. T., Jenkins, J. C., Mitchell, M. J., Rustad, L., & Haeuber, R. (2007). Who needs environmental monitoring? Frontiers in Ecology and the Environment, 5(5), 253–260.
30. Madakam, S., Ramaswamy, R., & Tripathi, S. (2015). Internet of Things (IoT): A literature review. Journal of Computer and Communications, 3(5), 164–173.
31. Mahmood, K., Risch, T., & Orsborn, K. (2021). Analytics of IIoT data using a NoSQL datastore. In IEEE International Conference on Smart Computing (SMARTCOMP) (pp. 97–104). IEEE.
32. Malasinghe, L. P., Ramzan, N., & Dahal, K. (2019). Remote patient monitoring: A comprehensive study. Journal of Ambient Intelligence and Humanized Computing, 10, 57–76.
33. Mehmood, N. Q., Culmone, R., & Mostarda, L. (2017). Modeling temporal aspects of sensor data for MongoDB NoSQL database. Journal of Big Data, 4(1), 8.
34. Memon, M. H., Kumar, W., Memon, A., Chowdhry, B. S., Aamir, M., & Kumar, P. (2016). Internet of Things (IoT) enabled smart animal farm. In 3rd International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 2067–2072). IEEE.
35. MERNIS. (2025). MERNIS Project. https://www.nvi.gov.tr/mernis (Accessed May 13, 2025)
36. Meta Platforms. (2025). React: A JavaScript library for building user interfaces. https://reactjs.org/ (Accessed March 11, 2025)
37. Meta Platforms. (2025a). Jest: Delightful JavaScript testing. https://jestjs.io/ (Accessed January 5, 2025)
38. Microsoft. (2025a). Entity Framework documentation. https://learn.microsoft.com/en-us/ef/ (Accessed February 12, 2025)
39. Microsoft. (2025b). Microsoft SQL Server documentation. https://learn.microsoft.com/en-us/sql/sql-server/ (Accessed February 12, 2025)
40. Microsoft. (2025c). MSBuild: Build engine for Microsoft Visual Studio. https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild (Accessed February 12, 2025)
41. Microsoft. (2025d). Official images for Microsoft SQL Server based on Ubuntu. https://hub.docker.com/r/microsoft/mssql-server (Accessed February 12, 2025)
42. Microsoft. (2025e). Visual Studio Code. https://code.visualstudio.com/ (Accessed February 12, 2025)
43. Nottingham, M. (2010). HTTP cache-control extensions for stale content (RFC 5861). https://datatracker.ietf.org/doc/html/rfc5861 (Accessed March 1, 2025)
44. Oikonomou, G., Duquennoy, S., Elsts, A., Eriksson, J., Tanaka, Y., & Tsiftes, N. (2022). The Contiki-NG open source operating system for next generation IoT devices. SoftwareX, 18, 101089.
45. Oke, A. E., & Arowoiya, V. A. (2021). Evaluation of internet of things (IoT) application areas for sustainable construction. Smart and Sustainable Built Environment, 10(3), 387–402.
46. OpenJS. (2025a). Node.js. https://nodejs.org/ (Accessed March 8, 2025)
47. OpenJS. (2025b). Webpack: A static module bundler for modern JavaScript applications. https://webpack.js.org/ (Accessed March 8, 2025)
48. Raja, P., & Bagwari, S. (2018). IoT based military assistance and surveillance. In International Conference on Intelligent Circuits and Systems (ICICS) (pp. 340–344). IEEE.
49. Redis. (2025). Redis: The open source in-memory data store. https://redis.io/ (Accessed February 27, 2025)
50. Robert, C. M. (2003). Agile software development: principles, patterns, and practices (1st ed.). Pearson Prentice Hall.
51. Shirvanian, N., Shams, M., & Rahmani, A. M. (2022). Internet of Things data management: A systematic literature review. International Journal of Communication Systems, 35(14), e5267.
52. Siow, E., Tiropanis, T., & Hall, W. (2018). Analytics for the internet of things: A survey. ACM Computing Surveys, 51(4), 1–36.
53. Tallat, R., Hawbani, A., Wang, X., Al-Dubai, A., Zhao, L., Liu, Z., & Alsamhi, S. H. (2023). Navigating industry 5.0. IEEE Communications Surveys & Tutorials, 26(2), 1080–1126.
54. Texas Instruments. (2025). CC1352R SimpleLink multiband wireless MCU LaunchPad development kit. https://www.ti.com/product/CC1352R (Accessed February 11, 2025)
55. Valecce, G., Strazzella, S., Radesca, A., & Grieco, L. A. (2019). Solarfertigation: Internet of Things architecture for smart agriculture. In IEEE International Conference on Communications Workshops (ICC Workshops) (pp. 1–6). IEEE.
56. Velásquez, W., Munoz–Arcentales, A., & Rodriguez, J. S. (2018). A case study: Ingestion analysis of WSN data in databases using Docker. In 1st International Conference on Computer Applications & Information Security (ICCAIS) (pp. 1–6). IEEE.
57. Verma, A., & Shukla, V. (2019). Analyzing the influence of IoT in tourism industry. In International Conference on Sustainable Computing in Science, Technology and Management (SUSCOM) (pp. 2083–2093). IEEE.
58. Winter, T., Thubert, P., Brandt, A., Hui, J., Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur, J. P., & Alexander, R. (2012). RPL: IPv6 routing protocol for low-power and lossy networks (RFC 6550). https://tools.ietf.org/html/rfc6550 (Accessed January 27, 2025)
59. xUnit. (2025). xUnit.net: A free, open source, community-focused unit testing tool. https://xunit.net (Accessed March 3, 2025)
60. Yilmaz, N. K., & Hazar, H. B. (2019). The rise of internet of things (IoT) and its applications in finance and accounting. PressAcad Procedia, 10(1), 32–35.
61. Zhang, Q., Liu, L., Pu, C., Dou, Q., Wu, L., & Zhou, W. (2018). A comparative study of containers and virtual machines in big data environment. In 11th IEEE International Conference on Cloud Computing (CLOUD) (pp. 178–185). IEEE.