Managing Large Data Sets with Floor ORM in Flutter

As mobile applications evolve, the need to handle large and complex datasets locally becomes increasingly important. While SQLite is a robust option for local data persistence, managing it directly can be cumbersome. Floor, a powerful and type-safe SQLite ORM for Flutter, simplifies the process of working with SQLite databases. This article explores how to effectively manage large datasets using Floor in Flutter applications, focusing on practical implementation and performance optimization techniques.

What is Floor ORM?

Floor is a Flutter SQLite ORM library that provides compile-time checks and simplifies database operations. By using Floor, you can define database entities, DAOs (Data Access Objects), and perform database migrations in a type-safe manner, making it easier to manage and maintain your data layer.

Why Use Floor ORM for Large Data Sets?

• Type Safety: Compile-time checks reduce runtime errors.
• Code Generation: Generates boilerplate code, reducing manual coding.
• Ease of Use: Simplified API for CRUD operations.
• Database Migrations: Easy management of database schema changes.
• Asynchronous Operations: Supports asynchronous database operations, crucial for UI responsiveness when dealing with large datasets.

Setting Up Floor ORM

Before diving into large data sets, let’s set up Floor in a Flutter project.

Step 1: Add Dependencies

Include the necessary Floor dependencies in your pubspec.yaml file:

dependencies:
  floor: ^1.4.2

dev_dependencies:
  floor_generator: ^1.4.2
  build_runner: ^2.4.6

Step 2: Create an Entity

Define a database entity by annotating a data class with @entity. This represents a table in your SQLite database. Here’s an example:

import 'package:floor/floor.dart';

@entity
class Item {
  @PrimaryKey(autoGenerate: true)
  final int? id;

  final String name;
  final String description;
  final double price;

  Item({
    this.id,
    required this.name,
    required this.description,
    required this.price,
  });
}

Step 3: Create a DAO (Data Access Object)

A DAO provides an abstraction layer for accessing the database. Define methods to query, insert, update, and delete data.

import 'package:floor/floor.dart';
import 'package:your_app_name/item.dart';

@dao
abstract class ItemDao {
  @Query('SELECT * FROM Item')
  Future> getAllItems();

  @Query('SELECT * FROM Item WHERE id = :id')
  Future getItemById(int id);

  @Insert(onConflict: OnConflictStrategy.replace)
  Future insertItem(Item item);

  @Update
  Future updateItem(Item item);

  @Delete
  Future deleteItem(Item item);
}

Step 4: Create a Database Class

Define an abstract class annotated with @Database. This class manages the database connection and exposes the DAOs.

import 'dart:async';
import 'package:floor/floor.dart';
import 'package:sqflite/sqflite.dart' as sqflite;
import 'package:your_app_name/item.dart';
import 'package:your_app_name/item_dao.dart';

part 'app_database.g.dart';

@Database(version = 1, entities = [Item])
abstract class AppDatabase extends FloorDatabase {
  ItemDao get itemDao;
}

Step 5: Build the Database

Generate the database code by running the following command in the terminal:

flutter pub run build_runner build

Step 6: Initialize the Database

Initialize the database in your application by using the generated _$AppDatabase class:

import 'package:your_app_name/app_database.dart';
import 'package:your_app_name/item.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  final database = await $FloorAppDatabase
      .databaseBuilder('app_database.db')
      .build();

  final itemDao = database.itemDao;
  final item = Item(name: 'Test Item', description: 'A test item', price: 99.99);
  await itemDao.insertItem(item);

  final allItems = await itemDao.getAllItems();
  print('All items: $allItems');
}

Managing Large Datasets with Floor

Now that Floor is set up, let’s focus on strategies for handling large datasets efficiently.

1. Pagination

Instead of loading the entire dataset at once, fetch data in smaller chunks or pages. Pagination is essential for improving performance and reducing memory usage.

Implement Pagination in DAO

Modify the DAO to include offset and limit parameters in your queries:

@dao
abstract class ItemDao {
  @Query('SELECT * FROM Item LIMIT :limit OFFSET :offset')
  Future> getItemsPaginated(int limit, int offset);

  // Other methods...
}

Implement Pagination in UI

Use a state management solution (like Provider, Riverpod, or BLoC) to manage pagination state in your UI:

import 'package:flutter/material.dart';
import 'package:provider/provider.dart';
import 'package:your_app_name/app_database.dart';
import 'package:your_app_name/item.dart';

class ItemProvider extends ChangeNotifier {
  final AppDatabase database;
  List _items = [];
  int _page = 0;
  final int _limit = 20;
  bool _isLoading = false;
  bool _hasMore = true;

  ItemProvider(this.database) {
    fetchItems();
  }

  List get items => _items;
  bool get isLoading => _isLoading;
  bool get hasMore => _hasMore;

  Future fetchItems() async {
    if (_isLoading || !_hasMore) return;

    _isLoading = true;
    notifyListeners();

    final itemDao = database.itemDao;
    final newItems = await itemDao.getItemsPaginated(_limit, _page * _limit);

    if (newItems.isEmpty) {
      _hasMore = false;
    } else {
      _items.addAll(newItems);
      _page++;
    }

    _isLoading = false;
    notifyListeners();
  }
}

class ItemListScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final itemProvider = Provider.of(context);

    return Scaffold(
      appBar: AppBar(title: Text('Items')),
      body: ListView.builder(
        itemCount: itemProvider.items.length + (itemProvider.hasMore ? 1 : 0),
        itemBuilder: (context, index) {
          if (index < itemProvider.items.length) {
            final item = itemProvider.items[index];
            return ListTile(
              title: Text(item.name),
              subtitle: Text(item.description),
            );
          } else if (itemProvider.hasMore) {
            itemProvider.fetchItems();
            return Center(child: CircularProgressIndicator());
          } else {
            return Container();
          }
        },
      ),
    );
  }
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  final database = await $FloorAppDatabase
      .databaseBuilder('app_database.db')
      .build();

  runApp(
    MaterialApp(
      home: ChangeNotifierProvider(
        create: (context) => ItemProvider(database),
        child: ItemListScreen(),
      ),
    ),
  );
}

2. Indexing

Creating indexes on frequently queried columns can significantly improve query performance, especially when dealing with large datasets.

Create Indexes in Entity Definition

Add indexes to the @entity annotation:

@entity(indices: [
  Index(value: ['name'], unique: false),
  Index(value: ['price'], unique: false),
])
class Item {
  @PrimaryKey(autoGenerate: true)
  final int? id;

  final String name;
  final String description;
  final double price;

  Item({
    this.id,
    required this.name,
    required this.description,
    required this.price,
  });
}

By defining indexes on the name and price columns, queries that filter or sort based on these columns will be significantly faster.

3. Asynchronous Operations

Always perform database operations asynchronously to prevent blocking the main thread and ensure a smooth user experience.

Floor ORM is Asynchronous by Default

Floor methods for database interactions, like queries, insertions, updates, and deletions, are asynchronous by default, returning Future objects.

4. Batch Processing

When inserting or updating a large number of records, use batch processing to group operations into a single transaction. This reduces the overhead of individual database operations.

Batch Insertion with Transactions

Use the database.transaction() method to perform batch operations:

Future insertItemsInBatch(List items) async {
  await database.transaction(() async {
    final itemDao = database.itemDao;
    for (final item in items) {
      await itemDao.insertItem(item);
    }
  });
}

5. Data Compression

If storage is a concern, consider compressing large text or binary data before storing it in the database.

6. Stream Data with Floor

Implement queries that return data as a stream for very large datasets, allowing incremental processing and UI updates.

import 'package:floor/floor.dart';
import 'package:your_app_name/item.dart';

@dao
abstract class ItemDao {
  @Query('SELECT * FROM Item')
  Stream> getAllItemsAsStream();

  @Insert(onConflict: OnConflictStrategy.replace)
  Future insertItem(Item item);

  // Other methods...
}

Consuming the Stream in UI

StreamBuilder>(
  stream: itemDao.getAllItemsAsStream(),
  builder: (BuildContext context, AsyncSnapshot> snapshot) {
    if (snapshot.hasData) {
      return ListView.builder(
        itemCount: snapshot.data!.length,
        itemBuilder: (context, index) {
          final item = snapshot.data![index];
          return ListTile(title: Text(item.name));
        },
      );
    } else if (snapshot.hasError) {
      return Text('Error: ${snapshot.error}');
    } else {
      return CircularProgressIndicator();
    }
  },
)

7. Optimization of Data Serialization

Ensure efficient serialization and deserialization when retrieving and storing large data sets to minimize overhead. Use efficient data types and avoid unnecessary data duplication.

Conclusion

Managing large datasets in Flutter applications using Floor ORM requires careful planning and optimization. By implementing techniques such as pagination, indexing, asynchronous operations, batch processing, and stream data, you can ensure that your application remains responsive and efficient even when dealing with substantial amounts of data. Floor simplifies these tasks with its type safety, code generation, and easy-to-use API, making it an excellent choice for Flutter developers. Always profile your application’s database performance to identify and address bottlenecks proactively.