How to make an R Shiny app with big data?

Question

I'm trying to create a simple R Shiny app that takes the chromosome, start, and end positions of CpGs as inputs and outputs boxplots for each CpG within the range. It works fine on the subset of data but fails for the real data ("all_data_reduced.RDS") because of its big size (almost 3 GB). I'm also not sure if the code written in the following way is the best, because even if the app works, it may take a lot of time to load from the user's end. Could you please suggest the best way to make an app when the data is big?

Please note that the users want it at the individual level. So, aggregating data to show only the means over the conditions (we have four conditions, two genders, and two types) isn't an option.

Also, I tried using the following option:

options(rsconnect.max.bundle.size = 4000000000)

#################################################################

library(shiny)
library(dplyr)
library(ggplot2)

#################################################################

# Load data
all_data_reduced <- readRDS("all_data_reduced.RDS")

#################################################################

# UI
ui <- fluidPage(
  titlePanel("Methylation Data Visualization for Cleft Lip and Palate (CP) Project"),
  sidebarLayout(
    sidebarPanel(
      selectInput("chromosome", "Chromosome:", choices = as.character(1:19)),
      numericInput("start_pos", "Start Position:", value = NULL),
      numericInput("end_pos", "End Position:", value = NULL)
    ),
    mainPanel(
      plotOutput("boxplots")
    )
  )
)


#################################################################

# Server
server <- function(input, output) {
  output$boxplots <- renderPlot({
    
    # Filtering dataset based on user defined inputs
    filtered_data <- subset(all_data_reduced, chr == input$chromosome & pos >= input$start_pos & pos <= input$end_pos)
    
    
    # Plot boxplots with facet_wrap for each chr_pos and overlay points
    ggplot(filtered_data, aes(x = Condition, y = methy_pct)) +
      geom_boxplot() +
      geom_point(position = position_jitter(width = 0.2), color = "red", size = 2) +  # Overlay points with jitter
      facet_wrap(~pos, scales = "free") +
      labs(title = "Methylation Percentage Boxplots", y = "Methylation %") +
      theme(
        plot.margin = margin(1, 1, 2, 1, "cm"),  # Adjust plot margin to increase plot size
        axis.text.x = element_text(angle = 45, hjust = 1)  # Angle x-axis labels
      ) +
      coord_cartesian(ylim = c(0, 1))  # Set y-axis limits within each facet
    
  })
}


#################################################################

# Run the application
shinyApp(ui = ui, server = server)