Posts: 23
Threads: 2
Joined: Oct 2023
This challenge is hella tough. I've tried everything and hit a wall. Got any hints or leads that help to crack it?
Posts: 1
Threads: 0
Joined: Jul 2024
I'm not sure i haven't solve it yet but you should try vector quantization method
Posts: 5
Threads: 1
Joined: Sep 2024
eqweqweqweqeqqqqqqqqqqqq This forum account is currently banned. Ban Length: Permanent (N/A Remaining)
Ban Reason: Leeching.
Posts: 4
Threads: 0
Joined: Jun 2024
Sep 06, 2024, 08:26 AM
(This post was last modified: Sep 06, 2024, 08:33 AM by nitwitse.)
One thing that made a big difference for me was to normalise the data first. The span on the Y axis is a lot bigger than for the X axis and this seemed to cause problems when looking for nearest neighbours.
I used this to get everything down to -1..1 on each axis
# Normalising the data for X, Y, Z
x_min, x_max = np.min(all_data[:, 0]), np.max(all_data[:, 0])
y_min, y_max = np.min(all_data[:, 1]), np.max(all_data[:, 1])
z_min, z_max = np.min(all_data[:, 2]), np.max(all_data[:, 2])
# Apply normalisation to each axis across the entire dataset
all_data[:, 0] = (all_data[:, 0] - x_min) / (x_max - x_min)
all_data[:, 1] = (all_data[:, 1] - y_min) / (y_max - y_min)
all_data[:, 2] = (all_data[:, 2] - z_min) / (z_max - z_min)
core_data[:, 0] = (core_data[:, 0] - x_min) / (x_max - x_min)
core_data[:, 1] = (core_data[:, 1] - y_min) / (y_max - y_min)
core_data[:, 2] = (core_data[:, 2] - z_min) / (z_max - z_min)
Oh, and visualising things is always critical for problems like this. The following code will give you a nice graph of your progress that you can zoom / rotate to find problem areas:
import numpy as np
import plotly.graph_objs as go
import plotly.io as pio
import plotly.colors as pc
valid_indices = np.where(np.where(out_labels != -1)[0])[0] # filter out the unabelled points
valid_labels = out_labels[valid_indices]
valid_stars = all_data[valid_indices]
# Separate core_data points from the valid stars
core_data_indices = np.arange(class_count)
core_data_stars = core_data # The original labelled stars
# Choose a qualitative colorscale for distinct colors
color_scale = pc.qualitative.Plotly # You can also try 'D3', 'Set1', etc.
# Create the 3D scatter plot for all stars
trace = go.Scatter3d(
x=valid_stars[:, 0],
y=valid_stars[:, 1],
z=valid_stars[:, 2],
mode='markers',
marker=dict(
size=1,
color=valid_labels, # Assign labels to colors
colorscale=color_scale, # Use the discrete colorscale
opacity=0.8,
colorbar=dict(title='Cluster Label')
),
text=[f"Index: {i}<br>Label: {label}<br>X: {x}<br>Y: {y}<br>Z: {z}"
for i, (x, y, z), label in zip(valid_indices, valid_stars, valid_labels)],
hoverinfo='text' # Display the custom text on hover
)
# Create another scatter plot for the core_data stars with text annotations
core_trace = go.Scatter3d(
x=core_data_stars[:, 0],
y=core_data_stars[:, 1],
z=core_data_stars[:, 2],
mode='markers+text',
marker=dict(
size=2,
color='red', # Different color for core_data points
opacity=0.9,
),
text=[str(i) for i in core_data_indices], # Display index as text
textposition='top center'
)
# Set up the layout with increased height
layout = go.Layout(
scene=dict(
xaxis=dict(title='X Coordinate'),
yaxis=dict(title='Y Coordinate'),
zaxis=dict(title='Z Coordinate')
),
width=1200, # You can adjust the width here
height=1200 # Increase the height here
)
# Create the figure with both traces
fig = go.Figure(data=[trace, core_trace], layout=layout)
# Render the plot in the notebook (or in a browser if not using a notebook environment)
pio.show(fig)
Posts: 10
Threads: 0
Joined: Sep 2024
I'm not sure i haven't solve it yet but you should try vector quantization method
Posts: 4
Threads: 0
Joined: Jun 2024
(Sep 06, 2024, 10:08 AM)dsk3kd Wrote: I'm not sure i haven't solve it yet but you should try vector quantization method
Unless I missed something with vector quantisation, it works on the assumption that the known points are the centroids for the clusters. Which in this dataset doesn't hold true for most of the clusters. So this approach might get you started with something half-way there, but I ended up going with a more manual and iterative approach to solve it: slowly growing the clusters iteratively by looking for nearest neighbours to the set of known points for each cluster, and slowly increasing the maximum search distance over time...
Would be great to know if there is a more elegant / off-the-shelf solution though!
Posts: 87
Threads: 28
Joined: Apr 2024
(Sep 06, 2024, 08:26 AM)nitwitse Wrote: One thing that made a big difference for me was to normalise the data first. The span on the Y axis is a lot bigger than for the X axis and this seemed to cause problems when looking for nearest neighbours.
I used this to get everything down to -1..1 on each axis
# Normalising the data for X, Y, Z
x_min, x_max = np.min(all_data[:, 0]), np.max(all_data[:, 0])
y_min, y_max = np.min(all_data[:, 1]), np.max(all_data[:, 1])
z_min, z_max = np.min(all_data[:, 2]), np.max(all_data[:, 2])
# Apply normalisation to each axis across the entire dataset
all_data[:, 0] = (all_data[:, 0] - x_min) / (x_max - x_min)
all_data[:, 1] = (all_data[:, 1] - y_min) / (y_max - y_min)
all_data[:, 2] = (all_data[:, 2] - z_min) / (z_max - z_min)
core_data[:, 0] = (core_data[:, 0] - x_min) / (x_max - x_min)
core_data[:, 1] = (core_data[:, 1] - y_min) / (y_max - y_min)
core_data[:, 2] = (core_data[:, 2] - z_min) / (z_max - z_min)
Oh, and visualising things is always critical for problems like this. The following code will give you a nice graph of your progress that you can zoom / rotate to find problem areas:
import numpy as np
import plotly.graph_objs as go
import plotly.io as pio
import plotly.colors as pc
valid_indices = np.where(np.where(out_labels != -1)[0])[0] # filter out the unabelled points
valid_labels = out_labels[valid_indices]
valid_stars = all_data[valid_indices]
# Separate core_data points from the valid stars
core_data_indices = np.arange(class_count)
core_data_stars = core_data # The original labelled stars
# Choose a qualitative colorscale for distinct colors
color_scale = pc.qualitative.Plotly # You can also try 'D3', 'Set1', etc.
# Create the 3D scatter plot for all stars
trace = go.Scatter3d(
x=valid_stars[:, 0],
y=valid_stars[:, 1],
z=valid_stars[:, 2],
mode='markers',
marker=dict(
size=1,
color=valid_labels, # Assign labels to colors
colorscale=color_scale, # Use the discrete colorscale
opacity=0.8,
colorbar=dict(title='Cluster Label')
),
text=[f"Index: {i}<br>Label: {label}<br>X: {x}<br>Y: {y}<br>Z: {z}"
for i, (x, y, z), label in zip(valid_indices, valid_stars, valid_labels)],
hoverinfo='text' # Display the custom text on hover
)
# Create another scatter plot for the core_data stars with text annotations
core_trace = go.Scatter3d(
x=core_data_stars[:, 0],
y=core_data_stars[:, 1],
z=core_data_stars[:, 2],
mode='markers+text',
marker=dict(
size=2,
color='red', # Different color for core_data points
opacity=0.9,
),
text=[str(i) for i in core_data_indices], # Display index as text
textposition='top center'
)
# Set up the layout with increased height
layout = go.Layout(
scene=dict(
xaxis=dict(title='X Coordinate'),
yaxis=dict(title='Y Coordinate'),
zaxis=dict(title='Z Coordinate')
),
width=1200, # You can adjust the width here
height=1200 # Increase the height here
)
# Create the figure with both traces
fig = go.Figure(data=[trace, core_trace], layout=layout)
# Render the plot in the notebook (or in a browser if not using a notebook environment)
pio.show(fig)
did you get the flag using this i tried to alter the code all i see is stars
Posts: 20
Threads: 4
Joined: Aug 2024
(Sep 06, 2024, 08:26 AM)nitwitse Wrote: One thing that made a big difference for me was to normalise the data first. The span on the Y axis is a lot bigger than for the X axis and this seemed to cause problems when looking for nearest neighbours.
I used this to get everything down to -1..1 on each axis
# Normalising the data for X, Y, Z
x_min, x_max = np.min(all_data[:, 0]), np.max(all_data[:, 0])
y_min, y_max = np.min(all_data[:, 1]), np.max(all_data[:, 1])
z_min, z_max = np.min(all_data[:, 2]), np.max(all_data[:, 2])
# Apply normalisation to each axis across the entire dataset
all_data[:, 0] = (all_data[:, 0] - x_min) / (x_max - x_min)
all_data[:, 1] = (all_data[:, 1] - y_min) / (y_max - y_min)
all_data[:, 2] = (all_data[:, 2] - z_min) / (z_max - z_min)
core_data[:, 0] = (core_data[:, 0] - x_min) / (x_max - x_min)
core_data[:, 1] = (core_data[:, 1] - y_min) / (y_max - y_min)
core_data[:, 2] = (core_data[:, 2] - z_min) / (z_max - z_min)
Oh, and visualising things is always critical for problems like this. The following code will give you a nice graph of your progress that you can zoom / rotate to find problem areas:
import numpy as np
import plotly.graph_objs as go
import plotly.io as pio
import plotly.colors as pc
valid_indices = np.where(np.where(out_labels != -1)[0])[0] # filter out the unabelled points
valid_labels = out_labels[valid_indices]
valid_stars = all_data[valid_indices]
# Separate core_data points from the valid stars
core_data_indices = np.arange(class_count)
core_data_stars = core_data # The original labelled stars
# Choose a qualitative colorscale for distinct colors
color_scale = pc.qualitative.Plotly # You can also try 'D3', 'Set1', etc.
# Create the 3D scatter plot for all stars
trace = go.Scatter3d(
x=valid_stars[:, 0],
y=valid_stars[:, 1],
z=valid_stars[:, 2],
mode='markers',
marker=dict(
size=1,
color=valid_labels, # Assign labels to colors
colorscale=color_scale, # Use the discrete colorscale
opacity=0.8,
colorbar=dict(title='Cluster Label')
),
text=[f"Index: {i}<br>Label: {label}<br>X: {x}<br>Y: {y}<br>Z: {z}"
for i, (x, y, z), label in zip(valid_indices, valid_stars, valid_labels)],
hoverinfo='text' # Display the custom text on hover
)
# Create another scatter plot for the core_data stars with text annotations
core_trace = go.Scatter3d(
x=core_data_stars[:, 0],
y=core_data_stars[:, 1],
z=core_data_stars[:, 2],
mode='markers+text',
marker=dict(
size=2,
color='red', # Different color for core_data points
opacity=0.9,
),
text=[str(i) for i in core_data_indices], # Display index as text
textposition='top center'
)
# Set up the layout with increased height
layout = go.Layout(
scene=dict(
xaxis=dict(title='X Coordinate'),
yaxis=dict(title='Y Coordinate'),
zaxis=dict(title='Z Coordinate')
),
width=1200, # You can adjust the width here
height=1200 # Increase the height here
)
# Create the figure with both traces
fig = go.Figure(data=[trace, core_trace], layout=layout)
# Render the plot in the notebook (or in a browser if not using a notebook environment)
pio.show(fig)
you will get flag from this code This forum account is currently banned. Ban Length: Permanent (N/A Remaining)
Ban Reason: Spamming (Copying other user's content) | https://breachforums.ai/Forum-Ban-Appeals if you feel this is incorrect.
Posts: 16
Threads: 0
Joined: Oct 2023
(Sep 06, 2024, 09:06 PM)09ft Wrote: (Sep 06, 2024, 08:26 AM)nitwitse Wrote: One thing that made a big difference for me was to normalise the data first. The span on the Y axis is a lot bigger than for the X axis and this seemed to cause problems when looking for nearest neighbours.
I used this to get everything down to -1..1 on each axis
# Normalising the data for X, Y, Z
x_min, x_max = np.min(all_data[:, 0]), np.max(all_data[:, 0])
y_min, y_max = np.min(all_data[:, 1]), np.max(all_data[:, 1])
z_min, z_max = np.min(all_data[:, 2]), np.max(all_data[:, 2])
# Apply normalisation to each axis across the entire dataset
all_data[:, 0] = (all_data[:, 0] - x_min) / (x_max - x_min)
all_data[:, 1] = (all_data[:, 1] - y_min) / (y_max - y_min)
all_data[:, 2] = (all_data[:, 2] - z_min) / (z_max - z_min)
core_data[:, 0] = (core_data[:, 0] - x_min) / (x_max - x_min)
core_data[:, 1] = (core_data[:, 1] - y_min) / (y_max - y_min)
core_data[:, 2] = (core_data[:, 2] - z_min) / (z_max - z_min)
Oh, and visualising things is always critical for problems like this. The following code will give you a nice graph of your progress that you can zoom / rotate to find problem areas:
import numpy as np
import plotly.graph_objs as go
import plotly.io as pio
import plotly.colors as pc
valid_indices = np.where(np.where(out_labels != -1)[0])[0] # filter out the unabelled points
valid_labels = out_labels[valid_indices]
valid_stars = all_data[valid_indices]
# Separate core_data points from the valid stars
core_data_indices = np.arange(class_count)
core_data_stars = core_data # The original labelled stars
# Choose a qualitative colorscale for distinct colors
color_scale = pc.qualitative.Plotly # You can also try 'D3', 'Set1', etc.
# Create the 3D scatter plot for all stars
trace = go.Scatter3d(
x=valid_stars[:, 0],
y=valid_stars[:, 1],
z=valid_stars[:, 2],
mode='markers',
marker=dict(
size=1,
color=valid_labels, # Assign labels to colors
colorscale=color_scale, # Use the discrete colorscale
opacity=0.8,
colorbar=dict(title='Cluster Label')
),
text=[f"Index: {i}<br>Label: {label}<br>X: {x}<br>Y: {y}<br>Z: {z}"
for i, (x, y, z), label in zip(valid_indices, valid_stars, valid_labels)],
hoverinfo='text' # Display the custom text on hover
)
# Create another scatter plot for the core_data stars with text annotations
core_trace = go.Scatter3d(
x=core_data_stars[:, 0],
y=core_data_stars[:, 1],
z=core_data_stars[:, 2],
mode='markers+text',
marker=dict(
size=2,
color='red', # Different color for core_data points
opacity=0.9,
),
text=[str(i) for i in core_data_indices], # Display index as text
textposition='top center'
)
# Set up the layout with increased height
layout = go.Layout(
scene=dict(
xaxis=dict(title='X Coordinate'),
yaxis=dict(title='Y Coordinate'),
zaxis=dict(title='Z Coordinate')
),
width=1200, # You can adjust the width here
height=1200 # Increase the height here
)
# Create the figure with both traces
fig = go.Figure(data=[trace, core_trace], layout=layout)
# Render the plot in the notebook (or in a browser if not using a notebook environment)
pio.show(fig)
you will get flag from this code
This does not work for me
|
