[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ANR for Firebase Performance #1810

Closed
carlonzo opened this issue Jul 23, 2020 · 13 comments
Closed

ANR for Firebase Performance #1810

carlonzo opened this issue Jul 23, 2020 · 13 comments
Assignees
Labels
api: performance api: remoteconfig needs-attention sdk-performance Issues where customers complain of an SDK/API being too slow

Comments

@carlonzo
Copy link
Contributor
carlonzo commented Jul 23, 2020

[READ] Step 1: Are you in the right place?

I do. reporting an ANR coming from RemoteConfigComponent (as far as I can say)

[REQUIRED] Step 2: Describe your environment

Firebase BoM 25.6.0 (Firebase Perf 19.0.8 if I'm not mistaken)

[REQUIRED] Step 3: Describe the problem

We are experiencing thousands ANRs at FirebasePerformance creation step. Looking at the source code, I believe the thread blocks trying to get/create an instance from RemoteConfigComponent.get("fireperf"). that method is access from main thread, is synchronized and internally it executes a few disk accesses.

Here is a stacktrace of the main thread and another one, both blocked.

Main thread:

at com.google.firebase.perf.zza.create (zza.java:2)
  at com.google.firebase.components.ComponentRuntime.lambda$new$0 (ComponentRuntime.java:69)
  at com.google.firebase.components.ComponentRuntime$$Lambda$1.get (ComponentRuntime.java)
  at com.google.firebase.components.Lazy.get (Lazy.java:53)
  at com.google.firebase.components.ComponentRuntime.initializeEagerComponents (ComponentRuntime.java:158)
  at com.google.firebase.FirebaseApp.initializeAllApis (FirebaseApp.java:563)
  at com.google.firebase.FirebaseApp.initializeApp (FirebaseApp.java:304)
  at com.google.firebase.FirebaseApp.initializeApp (FirebaseApp.java:268)
  at com.careem.superapp.core.analytics.impl.firebase.FirebaseInitializer.initialize (FirebaseInitializer.java:33)
  at com.careem.superapp.initialization.BaseSuperApp.onCreate (BaseSuperApp.java:58)
  at com.careem.superapp.SuperApp.onCreate (SuperApp.java:44)
  at android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1155)
  at android.app.ActivityThread.handleBindApplication (ActivityThread.java:6315)
  at android.app.ActivityThread.access$1300 (ActivityThread.java:220)
  at android.app.ActivityThread$H.handleMessage (ActivityThread.java:1815)
  at android.os.Handler.dispatchMessage (Handler.java:106)
  at android.os.Looper.loop (Looper.java:226)
  at android.app.ActivityThread.main (ActivityThread.java:7224)
  at java.lang.reflect.Method.invoke (Method.java)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:500)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:913)

Other thread:

at com.google.firebase.components.Lazy.get (SourceFile:3)
  at g.k.c.d.a.get (SourceFile:2)
  at com.google.firebase.components.ComponentRuntime.get (SourceFile:1)
  at com.google.firebase.FirebaseApp.get (SourceFile:2)
  at com.google.firebase.perf.FirebasePerformance.getInstance (SourceFile:5)
  at com.google.firebase.perf.internal.zzd.zzbt (SourceFile:2)
  at com.google.firebase.perf.internal.zzd.zzc (SourceFile:2)
  at g.k.c.k.a.b.run (SourceFile:1)
  at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
  at java.lang.Thread.run (Thread.java:764)

Steps to reproduce:

Sorry, can't repro locally. but the above stack trace should give an indication.
I don't think is relevant, but also wanted to add we disabled the FirebaseInitProvider and we init manually

Relevant Code:

@google-oss-bot
Copy link
Contributor

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

@ashwinraghav ashwinraghav added api: performance api: remoteconfig sdk-performance Issues where customers complain of an SDK/API being too slow labels Jul 23, 2020
@visumickey visumickey self-assigned this Jul 23, 2020
@visumickey
Copy link
Contributor

Internally tracked at b/162031027

@ramanpreetSinghKhinda
Copy link
Contributor

Hi carlonzo@,

I think this might help in further debugging we disabled the FirebaseInitProvider and we init manually. This looks to be a strange thread contention problem which may not always reproduce.

Context

Since FirebaseInitProvider initializes Firebase APIs at app startup time, not being able to initialize them properly (considering the inter-dependency) might cause conflicts.

From the logs it looks like com.careem.superapp.core.analytics.impl.firebase.FirebaseInitializer is the place where you are initializing FirebaseApp which is the entry point of Firebase SDKs (usually initialized by FirebaseInitProvider).

My hypothesis is that FirebaseApp initialization is stuck on initializing FirebasePerformance component because of the another thread in which FirebasePerformance.getInstance() seems to be called. As per the documentation of FirebaseApp:

Any FirebaseApp initialization must occur only in the main process of the app. Use of Firebase in processes other than the main process is not supported and will likely cause problems related to resource contention.

Remedies

  • Reverting back to use the FirebaseInitProvider (i.e not disabling it) may help to resolve this (but need to get more information before).

Questions

Q1. Is FirebaseInitializer a ContentProvider?

Q2. How and when are you calling FirebasePerformance.getInstance()?

Useful References

@google-oss-bot
Copy link
Contributor

Hey @carlonzo. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

@carlonzo
Copy link
Contributor Author
carlonzo commented Aug 3, 2020

Hi @ramanpreetSinghKhinda ,
Thanks for your help and sorry for my late answer.

I think I see what you are trying to say, but I cant find a reason we have this thread contention.

I can copy paste the content of that com.careem.superapp.core.analytics.impl.firebase.FirebaseInitializer

class FirebaseInitializer(
  private val isEnabled: Boolean,
  private val firebaseProjectConfig: FirebaseProjectConfig
) : InitializableAgent {

  private companion object {
    var isInitialized: Boolean = false
  }

  @Synchronized
  override fun initialize(context: Context) {
    if (!isInitialized && isEnabled) {

      val builder: FirebaseOptions.Builder
      with(firebaseProjectConfig) {
        builder = FirebaseOptions.Builder()
          .setApplicationId(googleAppId)
          .setApiKey(googleApiKey)
          .setDatabaseUrl(databaseUrl)
          .setStorageBucket(googleStorageBucket)
          .setProjectId(projectId)
          .setGcmSenderId(gcmDefaultSenderId)

        FirebaseApp.initializeApp(context, builder.build())
      }
      FirebaseAnalytics.getInstance(context).setAnalyticsCollectionEnabled(false)
    }
    isInitialized = true
  }
}

this is called inside the Application.onCreate. not from a Content provider. could that be the issue?

About your questions:
A1: no, FirebaseInitializer encapsulates the init method for FirebaseApp.initializeApp. pasted above.
A2: we are not manually calling FirebasePerformance.getInstance().

Thank you again for your time,
Carlo

@ramanpreetSinghKhinda
Copy link
Contributor
ramanpreetSinghKhinda commented Aug 4, 2020

Thanks @carlonzo

From the stack trace it does looks like FirebasePerformance.getInstance() is called. Can you instead let me know if you are calling any Fireperf specific API?

If yes, how are you calling that API? Any thread (or Synchronized) involved?

Just to rule that case out, your app is not a multi-process app, right?


Also just to clarify, you mentioned;

Looking at the source code, I believe the thread blocks trying to get/create an instance from RemoteConfigComponent.get("fireperf"). that method is access from main thread, is synchronized and internally it executes a few disk accesses

Can you point me out where exactly you looked?

@carlonzo
Copy link
Contributor Author
carlonzo commented Aug 5, 2020

Thanks again @ramanpreetSinghKhinda

No we do not directly call FirebasePerformance.getInstance(). we only apply the plugin. no call is done to any com.google.firebase.perf.*. we just use Firebase Performance to look at the default perfomance reports provided by the sdk (network, rendering by screen, .. ).

No our app is not a multi process app.

the reason I suggested was an issue with RemoteConfigComponent.get("fireperf") is bacause I tried to look at the content fo the class com.google.firebase.perf.zza.create (zza.java:2) (coming from the reported stacktrace) and the create method calls new FirebasePerformance((FirebaseApp)var1.get(FirebaseApp.class), ((RemoteConfigComponent)var1.get(RemoteConfigComponent.class)).get("fireperf")); (this is what I see from my decompiled class) and that RemoteConfigComponent.get("fireperf") is the only synchronized method I see. that methods loads a shared preference which can be blocking on IO. mine is just an hypothesis of course.

Please let me know if I can provide more infos

@ramanpreetSinghKhinda
Copy link
Contributor

Hi @carlonzo,

I'm still working on it. However, because of other priorities I am not yet able to spend time on validating this. As a remedy mentioned in #1810 (comment) can you revert to use the default way of Firebase initialization and see if that helps resolve your issue?

The reason for that suggestion (see this) is when an Android app process is first started, there is well-defined order of operations:

  1. Each ContentProvider declared in the manifest is created, in priority order.
  2. The Application class (or custom subclass) is created.
  3. If another component that was invoked via some Intent, that is created.

Since Firebase Initialization (in default case) happens in the ContentProvider, I suspect having it manually initialized is causing a problem in this particular case.

Note: In parallel we'll still keep on looking into why it's ANRing in this specialized use case.

@ramanpreetSinghKhinda
Copy link
Contributor

Hi @carlonzo,

I am looking into making a lazy dependency towards the RemoteConfigComponent, that should hopefully help resolve the thread contention problem.

If all goes well we'll release the fix in the upcoming release or next of that.

@carlonzo
Copy link
Contributor Author
carlonzo commented Sep 3, 2020

Hi @ramanpreetSinghKhinda thanks for that. We didn't switch to the initialisation using the content provider, yet, but I'm planning to do so this week. I'll let you know if I see improvements.

Hopefully making making the initialization lazy will fix the thread contention.

@ramanpreetSinghKhinda
Copy link
Contributor

@carlonzo

The fix will go in the upcoming release. You can keep an eye on the release notes for Firebase Performance.

Please reopen if the issue still persists with the latest perf sdk version.

@carlonzo
Copy link
Contributor Author

Thanks @ramanpreetSinghKhinda. looking forward to trying this.

Is there a public PR I can look at that fixed the issue?

@ramanpreetSinghKhinda
Copy link
Contributor
ramanpreetSinghKhinda commented Sep 16, 2020

Thanks for checking in @carlonzo

Not currently unfortunately :|

But once firebase-perf SDK gets open sourced you should be able to see the PRs and the code. This is currently in pipeline and will take a while. You can keep an eye on the Firebase Android Open Source Repo.

@firebase firebase locked and limited conversation to collaborators Oct 16, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
api: performance api: remoteconfig needs-attention sdk-performance Issues where customers complain of an SDK/API being too slow
Projects
None yet
Development

No branches or pull requests

5 participants