Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yggdrasil engine #256

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f84b0b1
Is enabled
gastonfournier Nov 16, 2023
ba5370c
All tests working
gastonfournier Nov 16, 2023
162d408
Removing shortcuts
gastonfournier Nov 16, 2023
48811ed
More cleanups
gastonfournier Nov 16, 2023
9a4a8ae
WIP
gastonfournier Nov 17, 2023
09ed04b
chore: fix some tests
gastonfournier Nov 17, 2023
3862610
Attempt to run tests with a compiled version of unleash-engine and yg…
gastonfournier Nov 17, 2023
c736c2b
Format changes
gastonfournier Nov 17, 2023
8baf2c6
chore: use published yggdrasil engine
sighphyre Nov 25, 2024
7674b6d
wip: move metrics to be handled by yggdrasil
sighphyre Nov 27, 2024
ddadd4d
chore: upgrade ygg to handle metrics bug
sighphyre Nov 27, 2024
8517b43
wip: repair metrics tests
sighphyre Nov 27, 2024
4ba1c45
chore: rename adapt context method
sighphyre Nov 27, 2024
d83d62d
fix client spec tests
sighphyre Nov 27, 2024
2ec31e5
chore: merge main
sighphyre Nov 28, 2024
0420d5c
chore: remove unneccessary stuff
sighphyre Nov 28, 2024
629636e
chore: remove ygg binary + jar, this is now resolved as a dependency
sighphyre Nov 28, 2024
2aec091
chore: restore some changes that just added noise
sighphyre Nov 28, 2024
12b5834
chore: remove dependent feature toggle test
sighphyre Nov 28, 2024
2a2e090
re enable test
sighphyre Nov 28, 2024
979229f
chore: trim out unneeded comment
sighphyre Nov 28, 2024
d49d244
chore: tease out yggdrasil adapters into their own static class
sighphyre Nov 28, 2024
cfefa56
chore: warn on fallback for unparsable dates
sighphyre Nov 28, 2024
76b99fa
Update src/main/java/io/getunleash/metric/UnleashMetricServiceImpl.java
sighphyre Nov 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@
</scm>

<dependencies>
<dependency>
<groupId>io.getunleash</groupId>
<artifactId>yggdrasil-engine</artifactId>
<version>0.1.0-alpha.9</version>
<classifier>x86_64-linux</classifier>
Copy link
Member Author

@sighphyre sighphyre Nov 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Classifier needs to be fixed before releasing this but I think that's a chunk of work distinct enough to want its own PR

</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
Expand Down
402 changes: 80 additions & 322 deletions src/main/java/io/getunleash/DefaultUnleash.java

Large diffs are not rendered by default.

21 changes: 0 additions & 21 deletions src/main/java/io/getunleash/FakeUnleash.java
Original file line number Diff line number Diff line change
Expand Up @@ -107,17 +107,6 @@ public void disableAllExcept(String... excludedFeatures) {
}
}

@Override
public Variant deprecatedGetVariant(String toggleName, UnleashContext context) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a public interface change. This was in place to support legacy evaluation of variants. We're removing that in the next major, as per #234

return null;
}

@Override
public Variant deprecatedGetVariant(
String toggleName, UnleashContext context, Variant defaultValue) {
return null;
}

public void resetAll() {
disableAll = false;
enableAll = false;
Expand Down Expand Up @@ -176,15 +165,5 @@ public List<EvaluatedToggle> evaluateAllToggles(@Nullable UnleashContext context
getVariant(toggleName)))
.collect(Collectors.toList());
}

@Override
public void count(String toggleName, boolean enabled) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Public interface change. I don't see a reason why anyone would be using this but it is public. The Yggdrasil engine is now the sink for this data

// Nothing to count
}

@Override
public void countVariant(String toggleName, String variantName) {
// Nothing to count
}
}
}
4 changes: 0 additions & 4 deletions src/main/java/io/getunleash/MoreOperations.java
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,4 @@ public interface MoreOperations {
* @return
*/
List<EvaluatedToggle> evaluateAllToggles(UnleashContext context);

void count(String toggleName, boolean enabled);

void countVariant(String toggleName, String variantName);
}
13 changes: 0 additions & 13 deletions src/main/java/io/getunleash/Unleash.java
Original file line number Diff line number Diff line change
Expand Up @@ -43,19 +43,6 @@ default Variant getVariant(final String toggleName, final Variant defaultValue)
return getVariant(toggleName, UnleashContext.builder().build(), defaultValue);
}

Variant deprecatedGetVariant(final String toggleName, final UnleashContext context);

Variant deprecatedGetVariant(
final String toggleName, final UnleashContext context, final Variant defaultValue);

default Variant deprecatedGetVariant(final String toggleName) {
return deprecatedGetVariant(toggleName, UnleashContext.builder().build());
}

default Variant deprecatedGetVariant(final String toggleName, final Variant defaultValue) {
return deprecatedGetVariant(toggleName, UnleashContext.builder().build(), defaultValue);
}

/**
* Use more().getFeatureToggleNames() instead
*
Expand Down
84 changes: 84 additions & 0 deletions src/main/java/io/getunleash/YggdrasilAdapters.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
package io.getunleash;

import io.getunleash.engine.Context;
import io.getunleash.engine.IStrategy;
import io.getunleash.engine.Payload;
import io.getunleash.engine.VariantDef;
import io.getunleash.lang.Nullable;
import io.getunleash.strategy.Strategy;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;
import java.util.Map;
import java.util.Optional;
import org.jetbrains.annotations.NotNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

final class YggdrasilAdapters {

private static final Logger LOGGER = LoggerFactory.getLogger(DefaultUnleash.class);

@NotNull
public static IStrategy adapt(Strategy s) {
return new IStrategy() {
@Override
public String getName() {
return s.getName();
}

@Override
public boolean isEnabled(Map<String, String> map, Context context) {
return s.isEnabled(map, adapt(context));
}
};
}

public static UnleashContext adapt(Context context) {
ZonedDateTime currentTime = ZonedDateTime.now();
if (context.getCurrentTime() != null) {
try {
currentTime = ZonedDateTime.parse(context.getCurrentTime());
} catch (DateTimeParseException e) {
LOGGER.warn("Could not parse current time from context, falling back to system time: ", context.getCurrentTime());
currentTime = ZonedDateTime.now();
}
}

return new UnleashContext(
context.getAppName(),
context.getEnvironment(),
context.getUserId(),
context.getSessionId(),
context.getRemoteAddress(),
currentTime,
context.getProperties());
}

public static Context adapt(UnleashContext context) {
Context mapped = new Context();
mapped.setAppName(context.getAppName().orElse(null));
mapped.setEnvironment(context.getEnvironment().orElse(null));
mapped.setUserId(context.getUserId().orElse(null));
mapped.setSessionId(context.getSessionId().orElse(null));
mapped.setRemoteAddress(context.getRemoteAddress().orElse(null));
mapped.setProperties(context.getProperties());
mapped.setCurrentTime(
DateTimeFormatter.ISO_DATE_TIME.format(
context.getCurrentTime().orElse(ZonedDateTime.now())));
return mapped;
}

public static Variant adapt(VariantDef variant, Variant defaultValue) {
if (variant == null) {
return defaultValue;
}
return new Variant(variant.getName(), adapt(variant.getPayload()), variant.isEnabled());
}

public static @Nullable io.getunleash.variant.Payload adapt(@Nullable Payload payload) {
return Optional.ofNullable(payload)
.map(p -> new io.getunleash.variant.Payload(p.getType(), p.getValue()))
.orElse(new io.getunleash.variant.Payload("string", null));
}
}
1 change: 1 addition & 0 deletions src/main/java/io/getunleash/metric/ClientMetrics.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package io.getunleash.metric;

import io.getunleash.engine.MetricsBucket;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not needed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its a bit sneaky but this was a drop in replacement for the existing metrics bucket. All I needed to do was include the new import and the code is unchanged

import io.getunleash.event.UnleashEvent;
import io.getunleash.event.UnleashSubscriber;
import io.getunleash.lang.Nullable;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,14 @@
import io.getunleash.event.EventDispatcher;
import io.getunleash.util.AtomicLongSerializer;
import io.getunleash.util.DateTimeSerializer;
import io.getunleash.util.InstantSerializer;
import io.getunleash.util.UnleashConfig;
import io.getunleash.util.UnleashURLs;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.time.Instant;
import java.time.LocalDateTime;
import java.util.concurrent.atomic.AtomicLong;

Expand All @@ -33,6 +35,7 @@ public DefaultHttpMetricsSender(UnleashConfig unleashConfig) {
this.gson =
new GsonBuilder()
.registerTypeAdapter(LocalDateTime.class, new DateTimeSerializer())
.registerTypeAdapter(Instant.class, new InstantSerializer())
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yggdrasil deals in Instants not Datetimes. This allows us to serialize those to ISO 8601 when we send metrics

.registerTypeAdapter(AtomicLong.class, new AtomicLongSerializer())
.create();
}
Expand Down
47 changes: 0 additions & 47 deletions src/main/java/io/getunleash/metric/MetricsBucket.java

This file was deleted.

4 changes: 0 additions & 4 deletions src/main/java/io/getunleash/metric/UnleashMetricService.java
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,4 @@

public interface UnleashMetricService {
void register(Set<String> strategies);

void count(String toggleName, boolean active);

void countVariant(String toggleName, String variantName);
}
55 changes: 29 additions & 26 deletions src/main/java/io/getunleash/metric/UnleashMetricServiceImpl.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
package io.getunleash.metric;

import io.getunleash.engine.MetricsBucket;
import io.getunleash.engine.UnleashEngine;
import io.getunleash.engine.YggdrasilError;
import io.getunleash.util.Throttler;
import io.getunleash.util.UnleashConfig;
import io.getunleash.util.UnleashScheduledExecutor;
Expand All @@ -15,21 +18,25 @@ public class UnleashMetricServiceImpl implements UnleashMetricService {
private final UnleashConfig unleashConfig;
private final MetricSender metricSender;

// mutable
private volatile MetricsBucket currentMetricsBucket;
// synchronization is handled in the engine itself
private final UnleashEngine engine;

private final Throttler throttler;

public UnleashMetricServiceImpl(
UnleashConfig unleashConfig, UnleashScheduledExecutor executor) {
this(unleashConfig, unleashConfig.getMetricSenderFactory().apply(unleashConfig), executor);
UnleashConfig unleashConfig, UnleashScheduledExecutor executor, UnleashEngine engine) {
this(
unleashConfig,
unleashConfig.getMetricSenderFactory().apply(unleashConfig),
executor,
engine);
}

public UnleashMetricServiceImpl(
UnleashConfig unleashConfig,
MetricSender metricSender,
UnleashScheduledExecutor executor) {
this.currentMetricsBucket = new MetricsBucket();
UnleashScheduledExecutor executor,
UnleashEngine engine) {
this.started = LocalDateTime.now(ZoneId.of("UTC"));
this.unleashConfig = unleashConfig;
this.metricSender = metricSender;
Expand All @@ -38,6 +45,7 @@ public UnleashMetricServiceImpl(
(int) unleashConfig.getSendMetricsInterval(),
300,
unleashConfig.getUnleashURLs().getClientMetricsURL());
this.engine = engine;
long metricsInterval = unleashConfig.getSendMetricsInterval();

executor.setInterval(sendMetrics(), metricsInterval, metricsInterval);
Expand All @@ -50,29 +58,24 @@ public void register(Set<String> strategies) {
metricSender.registerClient(registration);
}

@Override
public void count(String toggleName, boolean active) {
currentMetricsBucket.registerCount(toggleName, active);
}

@Override
public void countVariant(String toggleName, String variantName) {
currentMetricsBucket.registerCount(toggleName, variantName);
}

private Runnable sendMetrics() {
return () -> {
if (throttler.performAction()) {
MetricsBucket metricsBucket = this.currentMetricsBucket;
this.currentMetricsBucket = new MetricsBucket();
metricsBucket.end();
ClientMetrics metrics = new ClientMetrics(unleashConfig, metricsBucket);
int statusCode = metricSender.sendMetrics(metrics);
if (statusCode >= 200 && statusCode < 400) {
throttler.decrementFailureCountAndResetSkips();
}
if (statusCode >= 400) {
throttler.handleHttpErrorCodes(statusCode);
try {
MetricsBucket bucket = this.engine.getMetrics();

ClientMetrics metrics = new ClientMetrics(unleashConfig, bucket);
int statusCode = metricSender.sendMetrics(metrics);
if (statusCode >= 200 && statusCode < 400) {
throttler.decrementFailureCountAndResetSkips();
}
if (statusCode >= 400) {
throttler.handleHttpErrorCodes(statusCode);
}
} catch (YggdrasilError e) {
LOGGER.error(
"Failed to retrieve metrics from the engine, this is a serious error, please report it",
e);
}
} else {
throttler.skipped();
Expand Down
15 changes: 15 additions & 0 deletions src/main/java/io/getunleash/repository/FeatureRepository.java
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import io.getunleash.util.UnleashConfig;
import io.getunleash.util.UnleashScheduledExecutor;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
import java.util.function.Consumer;
import java.util.stream.Collectors;
Expand All @@ -24,6 +25,8 @@ public class FeatureRepository implements IFeatureRepository {
private final FeatureFetcher featureFetcher;
private final EventDispatcher eventDispatcher;

private List<Consumer<FeatureCollection>> consumers = new LinkedList<>();

private final Throttler throttler;

private FeatureCollection featureCollection;
Expand Down Expand Up @@ -116,6 +119,10 @@ private void initCollections(UnleashScheduledExecutor executor) {
}
}

public void addConsumer(Consumer<FeatureCollection> consumer) {
this.consumers.add(consumer);
}

private Runnable updateFeatures(final Consumer<UnleashException> handler) {
return () -> {
if (throttler.performAction()) {
Expand All @@ -131,6 +138,14 @@ private Runnable updateFeatures(final Consumer<UnleashException> handler) {
? segmentCollection
: new SegmentCollection(Collections.emptyList()));

consumers.forEach(
consumer -> {
try {
consumer.accept(featureCollection);
} catch (Exception e) {
LOGGER.error("Error when calling consumer {}", consumer, e);
}
});
featureBackupHandler.write(featureCollection);
} else if (response.getStatus() == ClientFeaturesResponse.Status.UNAVAILABLE) {
if (!ready && unleashConfig.isSynchronousFetchOnInitialisation()) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@

import io.getunleash.Segment;
import io.getunleash.lang.Nullable;
import java.util.function.Consumer;

public interface IFeatureRepository extends ToggleRepository {
@Nullable
Segment getSegment(Integer id);

void addConsumer(Consumer<FeatureCollection> consumer);
}
Loading